2023-07-04T18:34:57.668 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-04T18:34:57.670 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} --verbose --owner scheduled_vshankar@teuthology --name vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi -- /home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492/orig.config.yaml 2023-07-04T18:34:57.716 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-04T18:34:57.847 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492 branch: wip-vshankar-testing-20230704.040242 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} email: vshankar@redhat.com first_in_suite: false job_id: '7326492' kernel: client: sha1: distro flavor: default kdb: true sha1: 84139c18d56f518615369e8310ae4ecf07a36dcd ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing-20230704.040242 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20230704.040242 sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20230704.040242 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_207977f2e527a6f2822da19c25b874c901ff76c8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.tests_from_xfstests_dev teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-04_11:45:30 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-04T18:34:57.848 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_207977f2e527a6f2822da19c25b874c901ff76c8/qa; will attempt to use it 2023-07-04T18:34:57.848 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_207977f2e527a6f2822da19c25b874c901ff76c8/qa/tasks 2023-07-04T18:34:57.849 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-04T18:34:57.851 INFO:teuthology.task.internal:Checking packages... 2023-07-04T18:34:57.893 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '207977f2e527a6f2822da19c25b874c901ff76c8' 2023-07-04T18:34:57.894 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-04T18:34:57.894 INFO:teuthology.packaging:ref: None 2023-07-04T18:34:57.895 INFO:teuthology.packaging:tag: None 2023-07-04T18:34:57.895 INFO:teuthology.packaging:branch: wip-vshankar-testing-20230704.040242 2023-07-04T18:34:57.895 INFO:teuthology.packaging:sha1: 207977f2e527a6f2822da19c25b874c901ff76c8 2023-07-04T18:34:57.896 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20230704.040242 2023-07-04T18:34:58.043 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4734.g207977f2 2023-07-04T18:34:58.044 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-04T18:34:58.053 INFO:teuthology.task.internal:no buildpackages task found 2023-07-04T18:34:58.053 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-04T18:34:58.063 INFO:teuthology.task.internal:Saving configuration 2023-07-04T18:34:58.079 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-04T18:34:58.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-04T18:34:58.117 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-04 18:29:54.923483', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-04T18:34:58.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-04 18:29:54.925494', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-04T18:34:58.141 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-04T18:34:58.148 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-07-04T18:34:58.148 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-07-04T18:34:58.149 INFO:teuthology.run_tasks:Running task console_log... 2023-07-04T18:34:58.249 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f84b5b2c280>, signals=[15]) 2023-07-04T18:34:58.250 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-04T18:34:58.259 INFO:teuthology.task.internal:Opening connections... 2023-07-04T18:34:58.260 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:34:58.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-04T18:34:58.332 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:34:58.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-04T18:34:58.404 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-04T18:34:58.411 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2023-07-04T18:34:58.431 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2023-07-04T18:34:58.431 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2023-07-04T18:34:58.491 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux" 2023-07-04T18:34:58.491 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.6 (Ootpa)" 2023-07-04T18:34:58.491 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel" 2023-07-04T18:34:58.492 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora" 2023-07-04T18:34:58.492 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.6" 2023-07-04T18:34:58.492 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2023-07-04T18:34:58.493 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-04T18:34:58.493 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2023-07-04T18:34:58.493 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-04T18:34:58.493 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/" 2023-07-04T18:34:58.494 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-04T18:34:58.494 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-04T18:34:58.494 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:34:58.494 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-04T18:34:58.495 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-04T18:34:58.495 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-04T18:34:58.495 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-04T18:34:58.496 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2023-07-04T18:34:58.528 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2023-07-04T18:34:58.549 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2023-07-04T18:34:58.549 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2023-07-04T18:34:58.609 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Red Hat Enterprise Linux" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8.6 (Ootpa)" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:ID="rhel" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="fedora" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8.6" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-04T18:34:58.610 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.redhat.com/" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-04T18:34:58.611 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-04T18:34:58.612 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-04T18:34:58.612 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-04T18:34:58.612 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2023-07-04T18:34:58.634 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-04T18:34:58.644 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-04T18:34:58.651 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-04T18:34:58.651 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2023-07-04T18:34:58.654 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2023-07-04T18:34:58.672 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-04T18:34:58.684 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-04T18:34:58.685 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2023-07-04T18:34:58.713 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2023-07-04T18:34:58.740 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-04T18:34:58.805 INFO:teuthology.run_tasks:Running task kernel... 2023-07-04T18:34:58.822 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '84139c18d56f518615369e8310ae4ecf07a36dcd'} 2023-07-04T18:34:58.823 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-07-04T18:34:58.823 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-07-04T18:34:58.824 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-04T18:34:58.825 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-04T18:34:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:34:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:34:58.849 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2023-07-04T18:34:58.849 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2023-07-04T18:34:58.907 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-04T18:34:58.907 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-372.9.1.el8.x86_64 2023-07-04T18:34:58.907 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2023-07-04T18:34:58.908 INFO:teuthology.orchestra.run.smithi082.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-04T18:34:58.909 INFO:teuthology.task.kernel:Running kernel on smithi082: 4.18.0-372.9.1.el8.x86_64 2023-07-04T18:34:58.909 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2023-07-04T18:35:00.055 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:00.057 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2023-07-04T18:35:00.061 INFO:teuthology.orchestra.run.smithi082.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:Unable to read consumer identity 2023-07-04T18:35:03.861 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-04T18:35:03.903 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 3.5 kB 00:02 2023-07-04T18:35:06.278 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 16 MB 00:02 2023-07-04T18:35:06.387 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 16 MB 00:02 2023-07-04T18:35:09.678 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 447 kB/s | 24 kB 00:00 2023-07-04T18:35:09.803 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-07-04T18:35:10.372 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-04T18:35:10.422 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-07-04T18:35:10.423 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2023-07-04T18:35:10.423 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-07-04T18:35:10.447 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-04T18:35:10.497 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-07-04T18:35:10.498 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-07-04T18:35:10.498 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-07-04T18:35:10.574 DEBUG:teuthology.orchestra.run.smithi082:> echo no | sudo yum reinstall kernel || true 2023-07-04T18:35:10.603 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2023-07-04T18:35:10.904 INFO:teuthology.orchestra.run.smithi082.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:10.905 INFO:teuthology.orchestra.run.smithi082.stdout:Unable to read consumer identity 2023-07-04T18:35:10.931 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:10.932 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2023-07-04T18:35:11.158 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:02 ago on Tue 04 Jul 2023 06:35:09 PM UTC. 2023-07-04T18:35:11.188 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Tue 04 Jul 2023 06:35:09 PM UTC. 2023-07-04T18:35:11.198 INFO:teuthology.orchestra.run.smithi082.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-04T18:35:11.228 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-04T18:35:11.232 INFO:teuthology.orchestra.run.smithi082.stderr:Error: No packages marked for reinstall. 2023-07-04T18:35:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2023-07-04T18:35:11.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum reinstall -y kernel || true 2023-07-04T18:35:11.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2023-07-04T18:35:11.613 INFO:teuthology.orchestra.run.smithi082.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:11.613 INFO:teuthology.orchestra.run.smithi082.stdout:Unable to read consumer identity 2023-07-04T18:35:11.638 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:35:11.639 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2023-07-04T18:35:11.867 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:02 ago on Tue 04 Jul 2023 06:35:09 PM UTC. 2023-07-04T18:35:11.892 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Tue 04 Jul 2023 06:35:09 PM UTC. 2023-07-04T18:35:11.908 INFO:teuthology.orchestra.run.smithi082.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-04T18:35:11.931 INFO:teuthology.orchestra.run.smithi082.stderr:Error: No packages marked for reinstall. 2023-07-04T18:35:11.932 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-04T18:35:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2023-07-04T18:35:11.978 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2023-07-04T18:35:12.016 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2023-07-04T18:35:12.057 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-04T18:35:12.058 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-04T18:35:12.058 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-04T18:35:12.058 DEBUG:teuthology.parallel:result is None 2023-07-04T18:35:12.096 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-04T18:35:12.097 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-04T18:35:12.097 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-04T18:35:12.098 DEBUG:teuthology.parallel:result is None 2023-07-04T18:35:12.098 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-04T18:35:12.181 INFO:teuthology.task.internal:Creating test directory... 2023-07-04T18:35:12.181 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-04T18:35:12.184 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-04T18:35:12.201 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-04T18:35:12.232 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-04T18:35:12.238 INFO:teuthology.task.internal:Creating archive directory... 2023-07-04T18:35:12.239 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-04T18:35:12.243 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-04T18:35:12.278 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-04T18:35:12.305 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-04T18:35:12.305 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-04T18:35:12.309 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-04T18:35:12.345 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-04T18:35:12.355 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-04T18:35:12.358 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-04T18:35:12.370 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-04T18:35:12.372 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-04T18:35:12.436 INFO:teuthology.task.internal:Configuring sudo... 2023-07-04T18:35:12.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-04T18:35:12.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-04T18:35:12.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-04T18:35:12.524 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-04T18:35:12.525 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-04T18:35:12.529 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-04T18:35:12.547 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-04T18:35:12.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-04T18:35:12.685 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-04T18:35:12.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-04T18:35:12.809 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:35:12.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-04T18:35:12.888 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-04T18:35:12.934 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-04T18:35:12.989 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-04T18:35:13.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-04T18:35:13.114 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:35:13.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-04T18:35:13.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2023-07-04T18:35:13.183 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2023-07-04T18:35:13.240 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-04T18:35:13.281 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-04T18:35:13.627 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-04T18:35:13.636 INFO:teuthology.task.internal:Starting timer... 2023-07-04T18:35:13.637 INFO:teuthology.run_tasks:Running task pcp... 2023-07-04T18:35:13.646 INFO:teuthology.run_tasks:Running task selinux... 2023-07-04T18:35:13.655 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-04T18:35:13.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2023-07-04T18:35:13.709 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2023-07-04T18:35:13.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2023-07-04T18:35:13.762 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2023-07-04T18:35:13.766 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-04T18:35:13.767 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2023-07-04T18:35:13.792 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2023-07-04T18:35:13.792 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2023-07-04T18:35:13.819 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2023-07-04T18:35:13.819 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2023-07-04T18:35:13.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-04T18:35:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:35:13.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-04T18:35:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:35:13.887 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-04T18:35:13.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2023-07-04T18:35:13.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2023-07-04T18:35:13.953 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-04T18:35:13.962 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-04T18:35:13.969 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-04T18:35:14.121 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-04T18:35:14.142 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-04T18:35:14.144 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-04T18:43:48.262 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2023-07-04T18:43:48.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-07-04T18:43:48.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-04T18:43:48.334 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-07-04T18:43:48.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-07-04T18:43:48.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-07-04T18:43:48.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-04T18:43:48.471 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-07-04T18:43:48.538 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-07-04T18:43:48.538 INFO:teuthology.run_tasks:Running task clock... 2023-07-04T18:43:48.548 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-04T18:43:48.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-04T18:43:48.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-04T18:43:48.551 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-04T18:43:48.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-04T18:43:48.581 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-04T18:43:48.595 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-04T18:43:48.608 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-04T18:43:48.623 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-04T18:43:48.660 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2023-07-04T18:43:48.671 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2023-07-04T18:43:48.686 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-04T18:43:48.699 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-04T18:43:48.703 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2023-07-04T18:43:48.714 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2023-07-04T18:43:48.728 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-04T18:43:48.743 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-04T18:43:48.780 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-07-04T18:43:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-04T18:43:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-07-04T18:43:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.785 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.810 INFO:teuthology.orchestra.run.smithi082.stderr:bash: ntpq: command not found 2023-07-04T18:43:48.813 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-04T18:43:48.814 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2023-07-04T18:43:48.814 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.814 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.814 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.814 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-04T18:43:48.815 INFO:teuthology.run_tasks:Running task install... 2023-07-04T18:43:48.823 DEBUG:teuthology.task.install:project ceph 2023-07-04T18:43:48.824 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'}} 2023-07-04T18:43:48.825 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'} 2023-07-04T18:43:48.825 INFO:teuthology.task.install:Using flavor: default 2023-07-04T18:43:48.836 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-04T18:43:48.836 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-04T18:43:48.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8', 'tag': None, 'wait_for_package': False} 2023-07-04T18:43:48.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=207977f2e527a6f2822da19c25b874c901ff76c8 2023-07-04T18:43:48.840 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8', 'tag': None, 'wait_for_package': False} 2023-07-04T18:43:48.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=207977f2e527a6f2822da19c25b874c901ff76c8 2023-07-04T18:43:48.977 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/ 2023-07-04T18:43:48.978 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4734.g207977f2 2023-07-04T18:43:48.984 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/ 2023-07-04T18:43:48.984 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4734.g207977f2 2023-07-04T18:43:49.145 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-04T18:43:49.146 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:43:49.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-04T18:43:49.156 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230704.040242/207977f2e527a6f2822da19c25b874c901ff76c8/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-04T18:43:49.157 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:43:49.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-04T18:43:49.177 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-04T18:43:49.177 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/207977f2e527a6f2822da19c25b874c901ff76c8/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-04T18:43:49.186 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-04T18:43:49.186 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/207977f2e527a6f2822da19c25b874c901ff76c8/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-04T18:43:49.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-04T18:43:49.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-04T18:43:49.320 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-04T18:43:49.329 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-04T18:43:49.385 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2023-07-04T18:43:49.387 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2023-07-04T18:43:49.394 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2023-07-04T18:43:49.395 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2023-07-04T18:43:49.758 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:43:49.764 INFO:teuthology.orchestra.run.smithi082.stdout:Updating Subscription Management repositories. 2023-07-04T18:43:50.015 INFO:teuthology.orchestra.run.smithi077.stdout:44 files removed 2023-07-04T18:43:50.023 INFO:teuthology.orchestra.run.smithi082.stdout:44 files removed 2023-07-04T18:43:50.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-04T18:43:50.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-04T18:43:50.391 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:43:50.397 INFO:teuthology.orchestra.run.smithi082.stdout:Updating Subscription Management repositories. 2023-07-04T18:43:51.060 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 259 kB/s | 81 kB 00:00 2023-07-04T18:43:51.181 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 183 kB/s | 81 kB 00:00 2023-07-04T18:43:51.453 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-07-04T18:43:51.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-07-04T18:43:51.828 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-07-04T18:43:51.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2023-07-04T18:43:54.354 INFO:teuthology.orchestra.run.smithi082.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-04T18:43:54.391 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-07-04T18:45:18.487 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 190 kB/s | 16 MB 01:24 2023-07-04T18:45:21.876 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 483 kB/s | 24 kB 00:00 2023-07-04T18:45:22.778 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-07-04T18:45:30.932 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 54 MB/s | 47 MB 00:00 2023-07-04T18:45:41.793 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-04T18:45:41.794 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-04T18:45:41.808 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-04T18:45:41.809 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-04T18:45:41.810 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-04T18:45:41.812 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-04T18:45:41.813 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-04T18:45:41.816 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-04T18:45:41.817 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-04T18:45:41.949 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-07-04T18:45:41.961 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:18.0.0-4734.g207977f2.el8 ceph 6.4 k 2023-07-04T18:45:41.962 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:18.0.0-4734.g207977f2.el8 ceph 5.4 M 2023-07-04T18:45:41.963 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 849 k 2023-07-04T18:45:41.963 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4734.g207977f2.el8 ceph 165 k 2023-07-04T18:45:41.963 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:18.0.0-4734.g207977f2.el8 ceph 1.5 M 2023-07-04T18:45:41.964 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 129 k 2023-07-04T18:45:41.964 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 1.7 M 2023-07-04T18:45:41.964 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 7.4 M 2023-07-04T18:45:41.964 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 47 k 2023-07-04T18:45:41.965 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:18.0.0-4734.g207977f2.el8 ceph 13 M 2023-07-04T18:45:41.965 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:18.0.0-4734.g207977f2.el8 ceph 48 M 2023-07-04T18:45:41.965 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 258 k 2023-07-04T18:45:41.966 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 166 k 2023-07-04T18:45:41.966 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:18.0.0-4734.g207977f2.el8 ceph 258 k 2023-07-04T18:45:41.966 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 18 k 2023-07-04T18:45:41.966 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-04T18:45:41.967 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-04T18:45:41.967 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-04T18:45:41.967 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:18.0.0-4734.g207977f2.el8 ceph 31 k 2023-07-04T18:45:41.968 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 771 k 2023-07-04T18:45:41.968 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:18.0.0-4734.g207977f2.el8 ceph 139 k 2023-07-04T18:45:41.968 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-04T18:45:41.968 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:18.0.0-4734.g207977f2.el8 ceph 198 k 2023-07-04T18:45:41.969 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:18.0.0-4734.g207977f2.el8 ceph 370 k 2023-07-04T18:45:41.970 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 364 k 2023-07-04T18:45:41.970 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:18.0.0-4734.g207977f2.el8 ceph 107 k 2023-07-04T18:45:41.970 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 94 k 2023-07-04T18:45:41.970 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.4 M 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 179 k 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.7 M 2023-07-04T18:45:41.971 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.6 M 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:18.0.0-4734.g207977f2.el8 ceph 22 M 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 23 k 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:18.0.0-4734.g207977f2.el8 ceph 2.3 M 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 240 k 2023-07-04T18:45:41.972 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:18.0.0-4734.g207977f2.el8 ceph 4.6 M 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 18 M 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 15 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:18.0.0-4734.g207977f2.el8 ceph 24 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-04T18:45:41.973 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:18.0.0-4734.g207977f2.el8 ceph 137 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-04T18:45:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 473 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 6.7 M 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-04T18:45:41.975 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-04T18:45:41.976 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-04T18:45:41.976 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-04T18:45:41.976 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-04T18:45:41.976 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-04T18:45:41.976 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-04T18:45:41.977 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-04T18:45:41.977 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-04T18:45:41.977 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-04T18:45:41.977 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-04T18:45:41.978 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-04T18:45:41.978 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-04T18:45:41.978 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 45 k 2023-07-04T18:45:41.978 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:18.0.0-4734.g207977f2.el8 ceph 119 k 2023-07-04T18:45:41.979 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-04T18:45:41.979 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-04T18:45:41.979 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-04T18:45:41.979 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-04T18:45:41.980 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-04T18:45:41.980 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-04T18:45:41.980 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-04T18:45:41.981 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-04T18:45:41.981 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-04T18:45:41.981 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-04T18:45:41.981 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-04T18:45:41.982 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-04T18:45:41.982 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-04T18:45:41.982 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-04T18:45:41.982 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-04T18:45:41.983 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-04T18:45:41.983 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-04T18:45:41.983 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-04T18:45:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-04T18:45:41.985 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-04T18:45:41.986 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-04T18:45:41.987 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql 10 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-07-04T18:45:41.988 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================================================== 2023-07-04T18:45:41.989 INFO:teuthology.orchestra.run.smithi077.stdout:Install 128 Packages 2023-07-04T18:45:41.989 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 3 Packages 2023-07-04T18:45:41.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:45:41.989 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 205 M 2023-07-04T18:45:41.990 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-07-04T18:45:42.304 INFO:teuthology.orchestra.run.smithi077.stdout:(1/131): ceph-18.0.0-4734.g207977f2.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-07-04T18:45:42.613 INFO:teuthology.orchestra.run.smithi077.stdout:(2/131): ceph-fuse-18.0.0-4734.g207977f2.el8.x8 2.7 MB/s | 849 kB 00:00 2023-07-04T18:45:42.696 INFO:teuthology.orchestra.run.smithi077.stdout:(3/131): ceph-immutable-object-cache-18.0.0-473 1.9 MB/s | 165 kB 00:00 2023-07-04T18:45:42.830 INFO:teuthology.orchestra.run.smithi077.stdout:(4/131): ceph-base-18.0.0-4734.g207977f2.el8.x8 7.5 MB/s | 5.4 MB 00:00 2023-07-04T18:45:43.022 INFO:teuthology.orchestra.run.smithi077.stdout:(5/131): ceph-mgr-18.0.0-4734.g207977f2.el8.x86 8.1 MB/s | 1.5 MB 00:00 2023-07-04T18:45:43.189 INFO:teuthology.orchestra.run.smithi077.stdout:(6/131): ceph-mds-18.0.0-4734.g207977f2.el8.x86 4.6 MB/s | 2.3 MB 00:00 2023-07-04T18:45:43.514 INFO:teuthology.orchestra.run.smithi077.stdout:(7/131): ceph-mon-18.0.0-4734.g207977f2.el8.x86 9.4 MB/s | 4.6 MB 00:00 2023-07-04T18:45:44.081 INFO:teuthology.orchestra.run.smithi077.stdout:(8/131): ceph-common-18.0.0-4734.g207977f2.el8. 11 MB/s | 22 MB 00:01 2023-07-04T18:45:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:(9/131): ceph-selinux-18.0.0-4734.g207977f2.el8 293 kB/s | 24 kB 00:00 2023-07-04T18:45:45.156 INFO:teuthology.orchestra.run.smithi077.stdout:(10/131): ceph-radosgw-18.0.0-4734.g207977f2.el 8.2 MB/s | 13 MB 00:01 2023-07-04T18:45:45.206 INFO:teuthology.orchestra.run.smithi077.stdout:(11/131): cephfs-mirror-18.0.0-4734.g207977f2.e 5.1 MB/s | 258 kB 00:00 2023-07-04T18:45:45.248 INFO:teuthology.orchestra.run.smithi077.stdout:(12/131): libcephfs-devel-18.0.0-4734.g207977f2 748 kB/s | 31 kB 00:00 2023-07-04T18:45:45.357 INFO:teuthology.orchestra.run.smithi077.stdout:(13/131): libcephfs2-18.0.0-4734.g207977f2.el8. 6.9 MB/s | 771 kB 00:00 2023-07-04T18:45:45.415 INFO:teuthology.orchestra.run.smithi077.stdout:(14/131): libcephsqlite-18.0.0-4734.g207977f2.e 2.3 MB/s | 137 kB 00:00 2023-07-04T18:45:45.574 INFO:teuthology.orchestra.run.smithi077.stdout:(15/131): ceph-osd-18.0.0-4734.g207977f2.el8.x8 7.4 MB/s | 18 MB 00:02 2023-07-04T18:45:45.599 INFO:teuthology.orchestra.run.smithi077.stdout:(16/131): librados-devel-18.0.0-4734.g207977f2. 756 kB/s | 139 kB 00:00 2023-07-04T18:45:45.658 INFO:teuthology.orchestra.run.smithi077.stdout:(17/131): libradosstriper1-18.0.0-4734.g207977f 5.5 MB/s | 473 kB 00:00 2023-07-04T18:45:45.742 INFO:teuthology.orchestra.run.smithi077.stdout:(18/131): python3-ceph-argparse-18.0.0-4734.g20 538 kB/s | 45 kB 00:00 2023-07-04T18:45:45.825 INFO:teuthology.orchestra.run.smithi077.stdout:(19/131): python3-ceph-common-18.0.0-4734.g2079 1.4 MB/s | 119 kB 00:00 2023-07-04T18:45:45.909 INFO:teuthology.orchestra.run.smithi077.stdout:(20/131): python3-cephfs-18.0.0-4734.g207977f2. 2.3 MB/s | 198 kB 00:00 2023-07-04T18:45:46.001 INFO:teuthology.orchestra.run.smithi077.stdout:(21/131): python3-rados-18.0.0-4734.g207977f2.e 3.9 MB/s | 370 kB 00:00 2023-07-04T18:45:46.093 INFO:teuthology.orchestra.run.smithi077.stdout:(22/131): python3-rbd-18.0.0-4734.g207977f2.el8 3.9 MB/s | 364 kB 00:00 2023-07-04T18:45:46.177 INFO:teuthology.orchestra.run.smithi077.stdout:(23/131): python3-rgw-18.0.0-4734.g207977f2.el8 1.3 MB/s | 107 kB 00:00 2023-07-04T18:45:46.285 INFO:teuthology.orchestra.run.smithi077.stdout:(24/131): librgw2-18.0.0-4734.g207977f2.el8.x86 9.8 MB/s | 6.7 MB 00:00 2023-07-04T18:45:46.310 INFO:teuthology.orchestra.run.smithi077.stdout:(25/131): rbd-fuse-18.0.0-4734.g207977f2.el8.x8 707 kB/s | 94 kB 00:00 2023-07-04T18:45:46.386 INFO:teuthology.orchestra.run.smithi077.stdout:(26/131): rbd-nbd-18.0.0-4734.g207977f2.el8.x86 2.3 MB/s | 179 kB 00:00 2023-07-04T18:45:46.461 INFO:teuthology.orchestra.run.smithi077.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4734.g 308 kB/s | 23 kB 00:00 2023-07-04T18:45:46.545 INFO:teuthology.orchestra.run.smithi077.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4734.g207977f 1.5 MB/s | 129 kB 00:00 2023-07-04T18:45:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:(29/131): rbd-mirror-18.0.0-4734.g207977f2.el8. 10 MB/s | 3.4 MB 00:00 2023-07-04T18:45:46.779 INFO:teuthology.orchestra.run.smithi077.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4734.g20797 7.3 MB/s | 1.7 MB 00:00 2023-07-04T18:45:46.862 INFO:teuthology.orchestra.run.smithi077.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4734.g20 2.8 MB/s | 240 kB 00:00 2023-07-04T18:45:46.946 INFO:teuthology.orchestra.run.smithi077.stdout:(32/131): ceph-mgr-rook-18.0.0-4734.g207977f2.e 570 kB/s | 47 kB 00:00 2023-07-04T18:45:47.029 INFO:teuthology.orchestra.run.smithi077.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4734.g2 174 kB/s | 15 kB 00:00 2023-07-04T18:45:47.121 INFO:teuthology.orchestra.run.smithi077.stdout:(34/131): ceph-volume-18.0.0-4734.g207977f2.el8 2.8 MB/s | 258 kB 00:00 2023-07-04T18:45:47.205 INFO:teuthology.orchestra.run.smithi077.stdout:(35/131): cephadm-18.0.0-4734.g207977f2.el8.noa 2.0 MB/s | 166 kB 00:00 2023-07-04T18:45:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:(36/131): cephfs-top-18.0.0-4734.g207977f2.el8. 216 kB/s | 18 kB 00:00 2023-07-04T18:45:47.389 INFO:teuthology.orchestra.run.smithi077.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 9.6 MB/s | 7.4 MB 00:00 2023-07-04T18:45:47.566 INFO:teuthology.orchestra.run.smithi077.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 336 kB/s | 59 kB 00:00 2023-07-04T18:45:47.598 INFO:teuthology.orchestra.run.smithi077.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 992 kB/s | 306 kB 00:00 2023-07-04T18:45:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 750 kB/s | 75 kB 00:00 2023-07-04T18:45:47.700 INFO:teuthology.orchestra.run.smithi077.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.6 MB/s | 476 kB 00:00 2023-07-04T18:45:47.724 INFO:teuthology.orchestra.run.smithi077.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 746 kB/s | 44 kB 00:00 2023-07-04T18:45:47.766 INFO:teuthology.orchestra.run.smithi077.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-07-04T18:45:47.783 INFO:teuthology.orchestra.run.smithi077.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 563 kB/s | 33 kB 00:00 2023-07-04T18:45:47.824 INFO:teuthology.orchestra.run.smithi077.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 279 kB/s | 16 kB 00:00 2023-07-04T18:45:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2023-07-04T18:45:47.925 INFO:teuthology.orchestra.run.smithi077.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.8 MB/s | 384 kB 00:00 2023-07-04T18:45:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2023-07-04T18:45:47.976 INFO:teuthology.orchestra.run.smithi077.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2023-07-04T18:45:48.001 INFO:teuthology.orchestra.run.smithi077.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 349 kB/s | 18 kB 00:00 2023-07-04T18:45:48.051 INFO:teuthology.orchestra.run.smithi077.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 977 kB/s | 49 kB 00:00 2023-07-04T18:45:48.110 INFO:teuthology.orchestra.run.smithi077.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-07-04T18:45:48.177 INFO:teuthology.orchestra.run.smithi077.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 5.8 MB/s | 1.2 MB 00:00 2023-07-04T18:45:48.194 INFO:teuthology.orchestra.run.smithi077.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 722 kB/s | 60 kB 00:00 2023-07-04T18:45:48.244 INFO:teuthology.orchestra.run.smithi077.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2023-07-04T18:45:48.261 INFO:teuthology.orchestra.run.smithi077.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2023-07-04T18:45:48.295 INFO:teuthology.orchestra.run.smithi077.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 670 kB/s | 34 kB 00:00 2023-07-04T18:45:48.320 INFO:teuthology.orchestra.run.smithi077.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2023-07-04T18:45:48.354 INFO:teuthology.orchestra.run.smithi077.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2023-07-04T18:45:48.796 INFO:teuthology.orchestra.run.smithi077.stdout:(60/131): ceph-test-18.0.0-4734.g207977f2.el8.x 10 MB/s | 48 MB 00:04 2023-07-04T18:45:48.821 INFO:teuthology.orchestra.run.smithi077.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 37 kB/s | 19 kB 00:00 2023-07-04T18:45:48.846 INFO:teuthology.orchestra.run.smithi077.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 49 kB/s | 24 kB 00:00 2023-07-04T18:45:48.904 INFO:teuthology.orchestra.run.smithi077.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 473 kB/s | 27 kB 00:00 2023-07-04T18:45:48.947 INFO:teuthology.orchestra.run.smithi077.stdout:(64/131): python3-tempora-1.14.1-5.el8.noarch.r 194 kB/s | 29 kB 00:00 2023-07-04T18:45:48.980 INFO:teuthology.orchestra.run.smithi077.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.9 MB/s | 255 kB 00:00 2023-07-04T18:45:49.014 INFO:teuthology.orchestra.run.smithi077.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-07-04T18:45:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-07-04T18:45:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:(68/131): python3-websocket-client-0.56.0-5.el8 562 kB/s | 61 kB 00:00 2023-07-04T18:45:49.081 INFO:teuthology.orchestra.run.smithi077.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 341 kB/s | 23 kB 00:00 2023-07-04T18:45:49.106 INFO:teuthology.orchestra.run.smithi077.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-04T18:45:49.165 INFO:teuthology.orchestra.run.smithi077.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 562 kB/s | 47 kB 00:00 2023-07-04T18:45:49.190 INFO:teuthology.orchestra.run.smithi077.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.5 MB/s | 200 kB 00:00 2023-07-04T18:45:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 730 kB/s | 43 kB 00:00 2023-07-04T18:45:49.249 INFO:teuthology.orchestra.run.smithi077.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2023-07-04T18:45:49.274 INFO:teuthology.orchestra.run.smithi077.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 960 kB/s | 48 kB 00:00 2023-07-04T18:45:49.300 INFO:teuthology.orchestra.run.smithi077.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-07-04T18:45:49.325 INFO:teuthology.orchestra.run.smithi077.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 854 kB/s | 43 kB 00:00 2023-07-04T18:45:49.367 INFO:teuthology.orchestra.run.smithi077.stdout:(78/131): thrift-0.13.0-2.el8.x86_64.rpm 5.2 MB/s | 1.7 MB 00:00 2023-07-04T18:45:49.384 INFO:teuthology.orchestra.run.smithi077.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 1.3 MB/s | 109 kB 00:00 2023-07-04T18:45:49.417 INFO:teuthology.orchestra.run.smithi077.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2023-07-04T18:45:49.459 INFO:teuthology.orchestra.run.smithi077.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 7.3 MB/s | 560 kB 00:00 2023-07-04T18:45:49.485 INFO:teuthology.orchestra.run.smithi077.stdout:(82/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 186 kB/s | 19 kB 00:00 2023-07-04T18:45:49.502 INFO:teuthology.orchestra.run.smithi077.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.9 MB/s | 155 kB 00:00 2023-07-04T18:45:49.535 INFO:teuthology.orchestra.run.smithi077.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 7.2 MB/s | 559 kB 00:00 2023-07-04T18:45:49.561 INFO:teuthology.orchestra.run.smithi077.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 237 kB/s | 18 kB 00:00 2023-07-04T18:45:49.594 INFO:teuthology.orchestra.run.smithi077.stdout:(86/131): pcre2-devel-10.32-2.el8.x86_64.rpm 6.4 MB/s | 605 kB 00:00 2023-07-04T18:45:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-07-04T18:45:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-07-04T18:45:49.670 INFO:teuthology.orchestra.run.smithi077.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-07-04T18:45:49.729 INFO:teuthology.orchestra.run.smithi077.stdout:(90/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 14 MB/s | 2.3 MB 00:00 2023-07-04T18:45:49.754 INFO:teuthology.orchestra.run.smithi077.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 597 kB/s | 60 kB 00:00 2023-07-04T18:45:49.779 INFO:teuthology.orchestra.run.smithi077.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 903 kB/s | 98 kB 00:00 2023-07-04T18:45:49.804 INFO:teuthology.orchestra.run.smithi077.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 766 kB/s | 58 kB 00:00 2023-07-04T18:45:49.832 INFO:teuthology.orchestra.run.smithi077.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 512 kB/s | 39 kB 00:00 2023-07-04T18:45:49.863 INFO:teuthology.orchestra.run.smithi077.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 7.5 MB/s | 644 kB 00:00 2023-07-04T18:45:49.888 INFO:teuthology.orchestra.run.smithi077.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.0 MB/s | 171 kB 00:00 2023-07-04T18:45:49.914 INFO:teuthology.orchestra.run.smithi077.stdout:(97/131): NetworkManager-initscripts-updown-1.3 1.6 MB/s | 138 kB 00:00 2023-07-04T18:45:49.939 INFO:teuthology.orchestra.run.smithi077.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-07-04T18:45:49.956 INFO:teuthology.orchestra.run.smithi077.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-07-04T18:45:49.981 INFO:teuthology.orchestra.run.smithi077.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-04T18:45:50.015 INFO:teuthology.orchestra.run.smithi077.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-07-04T18:45:50.056 INFO:teuthology.orchestra.run.smithi077.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 516 kB/s | 39 kB 00:00 2023-07-04T18:45:50.090 INFO:teuthology.orchestra.run.smithi077.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 9.0 MB/s | 688 kB 00:00 2023-07-04T18:45:50.132 INFO:teuthology.orchestra.run.smithi077.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.0 MB/s | 538 kB 00:00 2023-07-04T18:45:50.157 INFO:teuthology.orchestra.run.smithi077.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-07-04T18:45:50.183 INFO:teuthology.orchestra.run.smithi077.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-04T18:45:50.216 INFO:teuthology.orchestra.run.smithi077.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-04T18:45:50.258 INFO:teuthology.orchestra.run.smithi077.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 641 kB/s | 27 kB 00:00 2023-07-04T18:45:50.308 INFO:teuthology.orchestra.run.smithi077.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-07-04T18:45:50.342 INFO:teuthology.orchestra.run.smithi077.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-07-04T18:45:50.426 INFO:teuthology.orchestra.run.smithi077.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-04T18:45:50.476 INFO:teuthology.orchestra.run.smithi077.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-04T18:45:50.518 INFO:teuthology.orchestra.run.smithi077.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-04T18:45:50.576 INFO:teuthology.orchestra.run.smithi077.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2023-07-04T18:45:50.602 INFO:teuthology.orchestra.run.smithi077.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 766 kB/s | 64 kB 00:00 2023-07-04T18:45:50.627 INFO:teuthology.orchestra.run.smithi077.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-04T18:45:50.652 INFO:teuthology.orchestra.run.smithi077.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 399 kB/s | 20 kB 00:00 2023-07-04T18:45:50.677 INFO:teuthology.orchestra.run.smithi077.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-07-04T18:45:50.711 INFO:teuthology.orchestra.run.smithi077.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-04T18:45:50.870 INFO:teuthology.orchestra.run.smithi077.stdout:(120/131): python3-scipy-1.0.0-21.module+el8.5. 14 MB/s | 14 MB 00:01 2023-07-04T18:45:50.895 INFO:teuthology.orchestra.run.smithi077.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-07-04T18:45:50.928 INFO:teuthology.orchestra.run.smithi077.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2023-07-04T18:45:50.971 INFO:teuthology.orchestra.run.smithi077.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-07-04T18:45:51.054 INFO:teuthology.orchestra.run.smithi077.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 13 MB/s | 4.9 MB 00:00 2023-07-04T18:45:51.138 INFO:teuthology.orchestra.run.smithi077.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-07-04T18:45:51.172 INFO:teuthology.orchestra.run.smithi077.stdout:(126/131): postgresql-contrib-10.21-2.module+el 6.7 MB/s | 808 kB 00:00 2023-07-04T18:45:51.230 INFO:teuthology.orchestra.run.smithi077.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 16 MB/s | 1.5 MB 00:00 2023-07-04T18:45:51.313 INFO:teuthology.orchestra.run.smithi077.stdout:(128/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-07-04T18:45:51.380 INFO:teuthology.orchestra.run.smithi077.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-07-04T18:45:51.714 INFO:teuthology.orchestra.run.smithi077.stdout:(130/131): librbd1-18.0.0-4734.g207977f2.el8.x8 7.4 MB/s | 3.6 MB 00:00 2023-07-04T18:45:51.781 INFO:teuthology.orchestra.run.smithi077.stdout:(131/131): librados2-18.0.0-4734.g207977f2.el8. 6.0 MB/s | 3.7 MB 00:00 2023-07-04T18:45:51.783 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-07-04T18:45:51.783 INFO:teuthology.orchestra.run.smithi077.stdout:Total 21 MB/s | 205 MB 00:09 2023-07-04T18:45:52.614 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-07-04T18:45:52.733 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-07-04T18:45:52.734 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-07-04T18:45:53.857 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-07-04T18:45:53.861 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-07-04T18:45:58.836 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-07-04T18:45:59.023 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-04T18:45:59.231 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-04T18:45:59.417 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-04T18:45:59.530 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-04T18:45:59.770 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-04T18:46:00.178 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-04T18:46:00.392 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:18.0.0-4734.g207977f2.el8.x86_64 5/134 2023-07-04T18:46:00.788 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:18.0.0-4734.g207977f2.el8.x86_64 5/134 2023-07-04T18:46:01.017 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:46:01.147 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:46:01.350 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:46:01.452 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:46:01.621 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 8/134 2023-07-04T18:46:01.825 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-04T18:46:02.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-04T18:46:02.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-04T18:46:02.977 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-04T18:46:03.228 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-04T18:46:03.417 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 14/134 2023-07-04T18:46:03.552 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 14/134 2023-07-04T18:46:03.741 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 15/134 2023-07-04T18:46:03.887 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 15/134 2023-07-04T18:46:04.087 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-04T18:46:04.272 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-04T18:46:04.451 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-04T18:46:04.583 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-04T18:46:04.718 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-04T18:46:04.850 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-04T18:46:05.449 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-04T18:46:05.600 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-04T18:46:06.199 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-04T18:46:06.383 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-04T18:46:07.078 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-04T18:46:07.346 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-04T18:46:07.522 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-04T18:46:07.742 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-04T18:46:07.897 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:18.0.0-4734.g207977f2.el8. 26/134 2023-07-04T18:46:08.295 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:18.0.0-4734.g207977f2.el 27/134 2023-07-04T18:46:08.544 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_6 28/134 2023-07-04T18:46:08.741 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-04T18:46:10.510 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-04T18:46:11.576 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-04T18:46:11.888 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-04T18:46:12.184 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-04T18:46:12.415 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-04T18:46:12.627 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-04T18:46:12.817 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-04T18:46:13.004 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 36/134 2023-07-04T18:46:13.163 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:18.0.0-4734.g207977f2.el8.x86_6 37/134 2023-07-04T18:46:13.331 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-04T18:46:13.492 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-04T18:46:14.015 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-04T18:46:14.084 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-04T18:46:14.335 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-04T18:46:14.594 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-04T18:46:14.774 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-04T18:46:14.853 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-04T18:46:15.054 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-04T18:46:15.211 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-04T18:46:15.394 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-04T18:46:15.592 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-04T18:46:15.801 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-04T18:46:15.952 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-04T18:46:16.173 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-04T18:46:16.381 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-04T18:46:16.739 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-04T18:46:17.033 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-04T18:46:17.242 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-04T18:46:17.493 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-04T18:46:17.736 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-04T18:46:18.081 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-04T18:46:18.339 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-04T18:46:18.515 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-04T18:46:18.737 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-04T18:46:18.930 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-04T18:46:19.111 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-04T18:46:19.425 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-04T18:46:19.720 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-04T18:46:19.977 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-04T18:46:20.216 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-04T18:46:20.364 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-04T18:46:20.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-04T18:46:20.791 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-04T18:46:21.021 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-04T18:46:21.169 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-04T18:46:21.325 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-04T18:46:21.531 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-04T18:46:21.734 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-04T18:46:21.925 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-04T18:46:22.126 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-04T18:46:22.287 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-04T18:46:22.504 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-04T18:46:22.709 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-04T18:46:22.904 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-04T18:46:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-04T18:46:23.422 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-04T18:46:23.606 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-04T18:46:23.834 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-04T18:46:24.930 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-04T18:46:25.176 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-04T18:46:25.422 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-04T18:46:25.714 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-04T18:46:26.036 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-04T18:46:26.228 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-04T18:46:27.019 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-04T18:46:27.197 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 92/134 2023-07-04T18:46:27.270 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 92/134 2023-07-04T18:46:27.414 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 93/134 2023-07-04T18:46:31.018 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:46:31.214 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:46:31.795 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:46:32.095 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 95/134 2023-07-04T18:46:32.180 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 95/134 2023-07-04T18:46:32.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 96/134 2023-07-04T18:46:50.956 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 96/134 2023-07-04T18:46:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2023-07-04T18:46:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2023-07-04T18:46:50.956 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2023-07-04T18:46:50.957 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-07-04T18:46:50.958 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:46:51.278 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 97/134 2023-07-04T18:46:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 97/134 2023-07-04T18:46:53.026 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 98/134 2023-07-04T18:46:53.245 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 98/134 2023-07-04T18:46:53.419 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 99/134 2023-07-04T18:46:53.885 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 99/134 2023-07-04T18:46:54.119 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 100/134 2023-07-04T18:46:54.136 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 100/134 2023-07-04T18:46:55.772 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:18.0.0-4734.g207977f2.el8.noarch 101/134 2023-07-04T18:46:55.920 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:18.0.0-4734.g207977f2.el8.noarch 101/134 2023-07-04T18:46:56.033 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.e 102/134 2023-07-04T18:46:56.207 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4734.g207977f2. 103/134 2023-07-04T18:46:56.393 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 104/134 2023-07-04T18:46:56.608 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 104/134 2023-07-04T18:46:56.919 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 105/134 2023-07-04T18:46:57.878 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 105/134 2023-07-04T18:46:58.026 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 106/134 2023-07-04T18:46:58.116 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 106/134 2023-07-04T18:46:58.470 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el 107/134 2023-07-04T18:46:58.667 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 108/134 2023-07-04T18:46:58.703 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 108/134 2023-07-04T18:46:58.876 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 109/134 2023-07-04T18:46:58.888 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 109/134 2023-07-04T18:47:00.223 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:18.0.0-4734.g207977f2.el8.x86_64 110/134 2023-07-04T18:47:00.438 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 111/134 2023-07-04T18:47:00.472 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 111/134 2023-07-04T18:47:00.643 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4734.g20797 112/134 2023-07-04T18:47:00.685 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4734.g20797 112/134 2023-07-04T18:47:00.818 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 113/134 2023-07-04T18:47:01.162 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 113/134 2023-07-04T18:47:01.338 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 114/134 2023-07-04T18:47:05.844 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 114/134 2023-07-04T18:47:06.717 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 115/134 2023-07-04T18:47:07.024 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-04T18:47:07.207 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 117/134 2023-07-04T18:47:07.449 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-04T18:47:07.609 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-04T18:47:07.799 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-04T18:47:08.038 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-04T18:47:08.221 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_ 121/134 2023-07-04T18:47:08.351 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-04T18:47:10.507 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:47:10.758 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:47:10.777 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:47:10.964 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 124/134 2023-07-04T18:47:11.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 125/134 2023-07-04T18:47:11.521 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 126/134 2023-07-04T18:47:11.887 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-04T18:47:12.083 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-04T18:47:12.168 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-04T18:47:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-04T18:47:12.413 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-04T18:47:12.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-04T18:47:12.574 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-04T18:47:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-04T18:47:12.702 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:47:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:47:12.895 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-04T18:47:13.114 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-04T18:47:13.315 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:47:13.802 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:47:13.802 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:18.0.0-4734.g207977f2.el8.x86_64 1/134 2023-07-04T18:47:13.802 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 2/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 3/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 4/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4734.g20797 5/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 8/134 2023-07-04T18:47:13.803 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 9/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 10/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 11/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 12/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 13/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_ 14/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 15/134 2023-07-04T18:47:13.804 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 16/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:18.0.0-4734.g207977f2.el8.x86_6 17/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 18/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 19/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4734.g207977f2.el 20/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:18.0.0-4734.g207977f2.el8. 21/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_6 22/134 2023-07-04T18:47:13.805 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 23/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 24/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 25/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 26/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 27/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 28/134 2023-07-04T18:47:13.806 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4734.g207977f2. 29/134 2023-07-04T18:47:13.818 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 30/134 2023-07-04T18:47:13.818 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 31/134 2023-07-04T18:47:13.819 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 32/134 2023-07-04T18:47:13.819 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el 33/134 2023-07-04T18:47:13.819 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 34/134 2023-07-04T18:47:13.819 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.e 35/134 2023-07-04T18:47:13.819 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 36/134 2023-07-04T18:47:13.820 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:18.0.0-4734.g207977f2.el8.noarch 37/134 2023-07-04T18:47:13.820 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 38/134 2023-07-04T18:47:13.820 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-04T18:47:13.821 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-04T18:47:13.821 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-04T18:47:13.821 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-04T18:47:13.822 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-04T18:47:13.822 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-04T18:47:13.822 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-04T18:47:13.822 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-04T18:47:13.823 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-04T18:47:13.824 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-04T18:47:13.825 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-04T18:47:13.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-04T18:47:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-04T18:47:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-04T18:47:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-04T18:47:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-04T18:47:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-04T18:47:13.828 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-04T18:47:13.828 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-04T18:47:13.828 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-04T18:47:13.829 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-04T18:47:13.830 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-04T18:47:13.830 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-04T18:47:13.830 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-04T18:47:13.830 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-04T18:47:13.831 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-04T18:47:13.832 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-04T18:47:13.833 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-04T18:47:13.833 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-04T18:47:13.833 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-04T18:47:13.833 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-04T18:47:13.834 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-04T18:47:13.834 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-04T18:47:13.834 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-04T18:47:13.835 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-04T18:47:13.835 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-04T18:47:13.835 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-04T18:47:13.835 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-04T18:47:13.836 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-04T18:47:13.836 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-04T18:47:13.836 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-04T18:47:13.837 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-04T18:47:13.837 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-04T18:47:13.837 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-04T18:47:13.838 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-04T18:47:13.838 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-04T18:47:13.838 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-04T18:47:13.838 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-04T18:47:13.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-04T18:47:13.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-04T18:47:13.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-04T18:47:13.840 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-04T18:47:13.840 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:18.0.0-4734.g207977f2.el8.x86_64 129/134 2023-07-04T18:47:13.840 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-04T18:47:13.840 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 131/134 2023-07-04T18:47:13.841 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:47:13.841 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-04T18:47:16.489 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:47:16.490 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:46 ago on Tue 04 Jul 2023 06:45:30 PM UTC. 2023-07-04T18:47:19.128 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2023-07-04T18:47:19.357 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:47:19.358 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-07-04T18:47:19.358 INFO:teuthology.orchestra.run.smithi077.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-04T18:47:19.358 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.358 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.1-7.el8.noarch 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.0.4-10.el8.x86_64 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.359 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.361 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-04T18:47:19.362 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.1-9.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-3.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.363 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: libpq-13.5-1.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-04T18:47:19.364 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-04T18:47:19.365 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-04T18:47:19.366 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-04T18:47:19.367 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-04T18:47:19.367 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-04T18:47:19.367 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:47:19.367 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:47:19.367 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.368 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-04T18:47:19.369 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-04T18:47:19.370 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-04T18:47:19.371 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-04T18:47:19.372 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-04T18:47:19.373 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-04T18:47:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-04T18:47:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-04T18:47:19.378 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-04T18:47:19.378 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-04T18:47:19.378 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-04T18:47:19.379 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-04T18:47:19.379 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-04T18:47:19.379 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-04T18:47:19.379 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-04T18:47:19.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-04T18:47:19.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-04T18:47:19.380 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.380 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.381 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:47:19.381 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-04T18:47:19.381 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-04T18:47:19.382 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-04T18:47:19.382 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-04T18:47:19.382 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-04T18:47:19.383 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-04T18:47:19.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:47:19.383 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-07-04T18:47:19.549 DEBUG:teuthology.parallel:result is None 2023-07-04T18:48:18.859 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 60 kB/s | 16 MB 04:24 2023-07-04T18:48:22.357 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 429 kB/s | 24 kB 00:00 2023-07-04T18:48:23.276 INFO:teuthology.orchestra.run.smithi082.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 47 MB 00:00 2023-07-04T18:48:31.375 INFO:teuthology.orchestra.run.smithi082.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 53 MB 00:01 2023-07-04T18:48:42.632 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-04T18:48:42.633 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-04T18:48:42.647 INFO:teuthology.orchestra.run.smithi082.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-04T18:48:42.648 INFO:teuthology.orchestra.run.smithi082.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-04T18:48:42.649 INFO:teuthology.orchestra.run.smithi082.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-04T18:48:42.651 INFO:teuthology.orchestra.run.smithi082.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-04T18:48:42.653 INFO:teuthology.orchestra.run.smithi082.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-04T18:48:42.656 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-04T18:48:42.657 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-04T18:48:42.788 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2023-07-04T18:48:42.801 INFO:teuthology.orchestra.run.smithi082.stdout:=========================================================================================================================== 2023-07-04T18:48:42.801 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2023-07-04T18:48:42.802 INFO:teuthology.orchestra.run.smithi082.stdout:=========================================================================================================================== 2023-07-04T18:48:42.803 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2023-07-04T18:48:42.803 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-04T18:48:42.803 INFO:teuthology.orchestra.run.smithi082.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-04T18:48:42.803 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:18.0.0-4734.g207977f2.el8 ceph 6.4 k 2023-07-04T18:48:42.804 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:18.0.0-4734.g207977f2.el8 ceph 5.4 M 2023-07-04T18:48:42.804 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 849 k 2023-07-04T18:48:42.804 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4734.g207977f2.el8 ceph 165 k 2023-07-04T18:48:42.805 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:18.0.0-4734.g207977f2.el8 ceph 1.5 M 2023-07-04T18:48:42.805 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 129 k 2023-07-04T18:48:42.805 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 1.7 M 2023-07-04T18:48:42.805 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 7.4 M 2023-07-04T18:48:42.806 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 47 k 2023-07-04T18:48:42.806 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:18.0.0-4734.g207977f2.el8 ceph 13 M 2023-07-04T18:48:42.807 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:18.0.0-4734.g207977f2.el8 ceph 48 M 2023-07-04T18:48:42.807 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 258 k 2023-07-04T18:48:42.807 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 166 k 2023-07-04T18:48:42.807 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror x86_64 2:18.0.0-4734.g207977f2.el8 ceph 258 k 2023-07-04T18:48:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 18 k 2023-07-04T18:48:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-04T18:48:42.808 INFO:teuthology.orchestra.run.smithi082.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-04T18:48:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-04T18:48:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:18.0.0-4734.g207977f2.el8 ceph 31 k 2023-07-04T18:48:42.809 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 771 k 2023-07-04T18:48:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:18.0.0-4734.g207977f2.el8 ceph 139 k 2023-07-04T18:48:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-04T18:48:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-04T18:48:42.810 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-04T18:48:42.811 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-04T18:48:42.811 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-04T18:48:42.812 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-04T18:48:42.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:18.0.0-4734.g207977f2.el8 ceph 198 k 2023-07-04T18:48:42.813 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:18.0.0-4734.g207977f2.el8 ceph 370 k 2023-07-04T18:48:42.813 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 364 k 2023-07-04T18:48:42.813 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:18.0.0-4734.g207977f2.el8 ceph 107 k 2023-07-04T18:48:42.814 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 94 k 2023-07-04T18:48:42.814 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.4 M 2023-07-04T18:48:42.814 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 179 k 2023-07-04T18:48:42.814 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-04T18:48:42.814 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.7 M 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 3.6 M 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:18.0.0-4734.g207977f2.el8 ceph 22 M 2023-07-04T18:48:42.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 23 k 2023-07-04T18:48:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:18.0.0-4734.g207977f2.el8 ceph 2.3 M 2023-07-04T18:48:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 240 k 2023-07-04T18:48:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:18.0.0-4734.g207977f2.el8 ceph 4.6 M 2023-07-04T18:48:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:18.0.0-4734.g207977f2.el8 ceph 18 M 2023-07-04T18:48:42.816 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4734.g207977f2.el8 ceph-noarch 15 k 2023-07-04T18:48:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:18.0.0-4734.g207977f2.el8 ceph 24 k 2023-07-04T18:48:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-04T18:48:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-04T18:48:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-04T18:48:42.817 INFO:teuthology.orchestra.run.smithi082.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:18.0.0-4734.g207977f2.el8 ceph 137 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-04T18:48:42.818 INFO:teuthology.orchestra.run.smithi082.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-04T18:48:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-04T18:48:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-04T18:48:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 473 k 2023-07-04T18:48:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-04T18:48:42.819 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:18.0.0-4734.g207977f2.el8 ceph 6.7 M 2023-07-04T18:48:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-04T18:48:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-04T18:48:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-04T18:48:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-04T18:48:42.820 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-04T18:48:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-04T18:48:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-04T18:48:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-04T18:48:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-04T18:48:42.821 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:18.0.0-4734.g207977f2.el8 ceph 45 k 2023-07-04T18:48:42.822 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:18.0.0-4734.g207977f2.el8 ceph 119 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-04T18:48:42.823 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-04T18:48:42.824 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-04T18:48:42.825 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-04T18:48:42.826 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-04T18:48:42.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-04T18:48:42.828 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-04T18:48:42.829 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-04T18:48:42.829 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-04T18:48:42.829 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-04T18:48:42.829 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-04T18:48:42.830 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-04T18:48:42.830 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-04T18:48:42.830 INFO:teuthology.orchestra.run.smithi082.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-04T18:48:42.830 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-04T18:48:42.831 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-04T18:48:42.831 INFO:teuthology.orchestra.run.smithi082.stdout:Enabling module streams: 2023-07-04T18:48:42.831 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql 10 2023-07-04T18:48:42.831 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:48:42.832 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2023-07-04T18:48:42.832 INFO:teuthology.orchestra.run.smithi082.stdout:=========================================================================================================================== 2023-07-04T18:48:42.832 INFO:teuthology.orchestra.run.smithi082.stdout:Install 128 Packages 2023-07-04T18:48:42.832 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 3 Packages 2023-07-04T18:48:42.833 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:48:42.833 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 205 M 2023-07-04T18:48:42.833 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2023-07-04T18:48:43.133 INFO:teuthology.orchestra.run.smithi082.stdout:(1/131): ceph-18.0.0-4734.g207977f2.el8.x86_64. 36 kB/s | 6.4 kB 00:00 2023-07-04T18:48:43.725 INFO:teuthology.orchestra.run.smithi082.stdout:(2/131): ceph-base-18.0.0-4734.g207977f2.el8.x8 7.0 MB/s | 5.4 MB 00:00 2023-07-04T18:48:43.783 INFO:teuthology.orchestra.run.smithi082.stdout:(3/131): ceph-immutable-object-cache-18.0.0-473 2.8 MB/s | 165 kB 00:00 2023-07-04T18:48:43.809 INFO:teuthology.orchestra.run.smithi082.stdout:(4/131): ceph-fuse-18.0.0-4734.g207977f2.el8.x8 1.2 MB/s | 849 kB 00:00 2023-07-04T18:48:44.009 INFO:teuthology.orchestra.run.smithi082.stdout:(5/131): ceph-mds-18.0.0-4734.g207977f2.el8.x86 10 MB/s | 2.3 MB 00:00 2023-07-04T18:48:44.426 INFO:teuthology.orchestra.run.smithi082.stdout:(6/131): ceph-mon-18.0.0-4734.g207977f2.el8.x86 11 MB/s | 4.6 MB 00:00 2023-07-04T18:48:44.660 INFO:teuthology.orchestra.run.smithi082.stdout:(7/131): ceph-common-18.0.0-4734.g207977f2.el8. 13 MB/s | 22 MB 00:01 2023-07-04T18:48:44.701 INFO:teuthology.orchestra.run.smithi082.stdout:(8/131): ceph-mgr-18.0.0-4734.g207977f2.el8.x86 1.7 MB/s | 1.5 MB 00:00 2023-07-04T18:48:44.785 INFO:teuthology.orchestra.run.smithi082.stdout:(9/131): ceph-selinux-18.0.0-4734.g207977f2.el8 292 kB/s | 24 kB 00:00 2023-07-04T18:48:45.852 INFO:teuthology.orchestra.run.smithi082.stdout:(10/131): ceph-osd-18.0.0-4734.g207977f2.el8.x8 12 MB/s | 18 MB 00:01 2023-07-04T18:48:46.010 INFO:teuthology.orchestra.run.smithi082.stdout:(11/131): ceph-radosgw-18.0.0-4734.g207977f2.el 10 MB/s | 13 MB 00:01 2023-07-04T18:48:46.037 INFO:teuthology.orchestra.run.smithi082.stdout:(12/131): cephfs-mirror-18.0.0-4734.g207977f2.e 1.4 MB/s | 258 kB 00:00 2023-07-04T18:48:46.065 INFO:teuthology.orchestra.run.smithi082.stdout:(13/131): libcephfs-devel-18.0.0-4734.g207977f2 619 kB/s | 31 kB 00:00 2023-07-04T18:48:46.103 INFO:teuthology.orchestra.run.smithi082.stdout:(14/131): libcephfs2-18.0.0-4734.g207977f2.el8. 11 MB/s | 771 kB 00:00 2023-07-04T18:48:46.132 INFO:teuthology.orchestra.run.smithi082.stdout:(15/131): libcephsqlite-18.0.0-4734.g207977f2.e 2.0 MB/s | 137 kB 00:00 2023-07-04T18:48:46.154 INFO:teuthology.orchestra.run.smithi082.stdout:(16/131): librados-devel-18.0.0-4734.g207977f2. 2.7 MB/s | 139 kB 00:00 2023-07-04T18:48:46.188 INFO:teuthology.orchestra.run.smithi082.stdout:(17/131): libradosstriper1-18.0.0-4734.g207977f 7.9 MB/s | 473 kB 00:00 2023-07-04T18:48:46.255 INFO:teuthology.orchestra.run.smithi082.stdout:(18/131): python3-ceph-argparse-18.0.0-4734.g20 672 kB/s | 45 kB 00:00 2023-07-04T18:48:46.330 INFO:teuthology.orchestra.run.smithi082.stdout:(19/131): python3-ceph-common-18.0.0-4734.g2079 1.5 MB/s | 119 kB 00:00 2023-07-04T18:48:46.413 INFO:teuthology.orchestra.run.smithi082.stdout:(20/131): python3-cephfs-18.0.0-4734.g207977f2. 2.3 MB/s | 198 kB 00:00 2023-07-04T18:48:46.505 INFO:teuthology.orchestra.run.smithi082.stdout:(21/131): python3-rados-18.0.0-4734.g207977f2.e 3.9 MB/s | 370 kB 00:00 2023-07-04T18:48:46.597 INFO:teuthology.orchestra.run.smithi082.stdout:(22/131): librgw2-18.0.0-4734.g207977f2.el8.x86 15 MB/s | 6.7 MB 00:00 2023-07-04T18:48:46.622 INFO:teuthology.orchestra.run.smithi082.stdout:(23/131): python3-rbd-18.0.0-4734.g207977f2.el8 3.0 MB/s | 364 kB 00:00 2023-07-04T18:48:46.647 INFO:teuthology.orchestra.run.smithi082.stdout:(24/131): python3-rgw-18.0.0-4734.g207977f2.el8 2.1 MB/s | 107 kB 00:00 2023-07-04T18:48:46.673 INFO:teuthology.orchestra.run.smithi082.stdout:(25/131): rbd-fuse-18.0.0-4734.g207977f2.el8.x8 1.8 MB/s | 94 kB 00:00 2023-07-04T18:48:46.773 INFO:teuthology.orchestra.run.smithi082.stdout:(26/131): rbd-nbd-18.0.0-4734.g207977f2.el8.x86 1.8 MB/s | 179 kB 00:00 2023-07-04T18:48:46.857 INFO:teuthology.orchestra.run.smithi082.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4734.g 276 kB/s | 23 kB 00:00 2023-07-04T18:48:46.916 INFO:teuthology.orchestra.run.smithi082.stdout:(28/131): rbd-mirror-18.0.0-4734.g207977f2.el8. 13 MB/s | 3.4 MB 00:00 2023-07-04T18:48:46.941 INFO:teuthology.orchestra.run.smithi082.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4734.g207977f 1.5 MB/s | 129 kB 00:00 2023-07-04T18:48:47.141 INFO:teuthology.orchestra.run.smithi082.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4734.g20797 7.6 MB/s | 1.7 MB 00:00 2023-07-04T18:48:47.225 INFO:teuthology.orchestra.run.smithi082.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4734.g20 2.8 MB/s | 240 kB 00:00 2023-07-04T18:48:47.292 INFO:teuthology.orchestra.run.smithi082.stdout:(32/131): ceph-mgr-rook-18.0.0-4734.g207977f2.e 711 kB/s | 47 kB 00:00 2023-07-04T18:48:47.359 INFO:teuthology.orchestra.run.smithi082.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4734.g2 218 kB/s | 15 kB 00:00 2023-07-04T18:48:47.492 INFO:teuthology.orchestra.run.smithi082.stdout:(34/131): ceph-mgr-diskprediction-local-18.0.0- 13 MB/s | 7.4 MB 00:00 2023-07-04T18:48:47.509 INFO:teuthology.orchestra.run.smithi082.stdout:(35/131): ceph-volume-18.0.0-4734.g207977f2.el8 1.7 MB/s | 258 kB 00:00 2023-07-04T18:48:47.535 INFO:teuthology.orchestra.run.smithi082.stdout:(36/131): cephadm-18.0.0-4734.g207977f2.el8.noa 3.9 MB/s | 166 kB 00:00 2023-07-04T18:48:47.551 INFO:teuthology.orchestra.run.smithi082.stdout:(37/131): cephfs-top-18.0.0-4734.g207977f2.el8. 427 kB/s | 18 kB 00:00 2023-07-04T18:48:47.643 INFO:teuthology.orchestra.run.smithi082.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 644 kB/s | 59 kB 00:00 2023-07-04T18:48:47.685 INFO:teuthology.orchestra.run.smithi082.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2023-07-04T18:48:47.711 INFO:teuthology.orchestra.run.smithi082.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-07-04T18:48:47.753 INFO:teuthology.orchestra.run.smithi082.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 6.9 MB/s | 476 kB 00:00 2023-07-04T18:48:47.778 INFO:teuthology.orchestra.run.smithi082.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 652 kB/s | 44 kB 00:00 2023-07-04T18:48:47.804 INFO:teuthology.orchestra.run.smithi082.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-07-04T18:48:47.837 INFO:teuthology.orchestra.run.smithi082.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 562 kB/s | 33 kB 00:00 2023-07-04T18:48:47.862 INFO:teuthology.orchestra.run.smithi082.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2023-07-04T18:48:47.896 INFO:teuthology.orchestra.run.smithi082.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-07-04T18:48:47.921 INFO:teuthology.orchestra.run.smithi082.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2023-07-04T18:48:47.946 INFO:teuthology.orchestra.run.smithi082.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-07-04T18:48:47.971 INFO:teuthology.orchestra.run.smithi082.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-07-04T18:48:47.999 INFO:teuthology.orchestra.run.smithi082.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-07-04T18:48:48.039 INFO:teuthology.orchestra.run.smithi082.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-07-04T18:48:48.072 INFO:teuthology.orchestra.run.smithi082.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 12 MB/s | 1.2 MB 00:00 2023-07-04T18:48:48.098 INFO:teuthology.orchestra.run.smithi082.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-07-04T18:48:48.123 INFO:teuthology.orchestra.run.smithi082.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-07-04T18:48:48.149 INFO:teuthology.orchestra.run.smithi082.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-07-04T18:48:48.174 INFO:teuthology.orchestra.run.smithi082.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-07-04T18:48:48.199 INFO:teuthology.orchestra.run.smithi082.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2023-07-04T18:48:48.225 INFO:teuthology.orchestra.run.smithi082.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-07-04T18:48:48.250 INFO:teuthology.orchestra.run.smithi082.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-07-04T18:48:48.275 INFO:teuthology.orchestra.run.smithi082.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 373 kB/s | 19 kB 00:00 2023-07-04T18:48:48.300 INFO:teuthology.orchestra.run.smithi082.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-07-04T18:48:48.326 INFO:teuthology.orchestra.run.smithi082.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2023-07-04T18:48:48.351 INFO:teuthology.orchestra.run.smithi082.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2023-07-04T18:48:48.376 INFO:teuthology.orchestra.run.smithi082.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-07-04T18:48:48.435 INFO:teuthology.orchestra.run.smithi082.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2023-07-04T18:48:48.468 INFO:teuthology.orchestra.run.smithi082.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 667 kB/s | 61 kB 00:00 2023-07-04T18:48:48.494 INFO:teuthology.orchestra.run.smithi082.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2023-07-04T18:48:48.519 INFO:teuthology.orchestra.run.smithi082.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 458 kB/s | 23 kB 00:00 2023-07-04T18:48:48.544 INFO:teuthology.orchestra.run.smithi082.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-04T18:48:48.586 INFO:teuthology.orchestra.run.smithi082.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2023-07-04T18:48:48.645 INFO:teuthology.orchestra.run.smithi082.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.0 MB/s | 110 kB 00:00 2023-07-04T18:48:48.679 INFO:teuthology.orchestra.run.smithi082.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-07-04T18:48:48.720 INFO:teuthology.orchestra.run.smithi082.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-04T18:48:48.762 INFO:teuthology.orchestra.run.smithi082.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-04T18:48:48.796 INFO:teuthology.orchestra.run.smithi082.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-07-04T18:48:48.846 INFO:teuthology.orchestra.run.smithi082.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2023-07-04T18:48:48.896 INFO:teuthology.orchestra.run.smithi082.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 10 MB/s | 538 kB 00:00 2023-07-04T18:48:48.938 INFO:teuthology.orchestra.run.smithi082.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-04T18:48:49.081 INFO:teuthology.orchestra.run.smithi082.stdout:(79/131): python3-scipy-1.0.0-21.module+el8.5.0 26 MB/s | 14 MB 00:00 2023-07-04T18:48:49.114 INFO:teuthology.orchestra.run.smithi082.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-07-04T18:48:49.156 INFO:teuthology.orchestra.run.smithi082.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-07-04T18:48:49.190 INFO:teuthology.orchestra.run.smithi082.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 796 kB/s | 27 kB 00:00 2023-07-04T18:48:49.252 INFO:teuthology.orchestra.run.smithi082.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 5.0 MB/s | 320 kB 00:00 2023-07-04T18:48:49.290 INFO:teuthology.orchestra.run.smithi082.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.1 MB/s | 44 kB 00:00 2023-07-04T18:48:49.366 INFO:teuthology.orchestra.run.smithi082.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-04T18:48:49.408 INFO:teuthology.orchestra.run.smithi082.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-07-04T18:48:49.449 INFO:teuthology.orchestra.run.smithi082.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-07-04T18:48:49.517 INFO:teuthology.orchestra.run.smithi082.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 17 MB/s | 3.7 MB 00:00 2023-07-04T18:48:49.542 INFO:teuthology.orchestra.run.smithi082.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 696 kB/s | 64 kB 00:00 2023-07-04T18:48:49.568 INFO:teuthology.orchestra.run.smithi082.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-07-04T18:48:49.593 INFO:teuthology.orchestra.run.smithi082.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 396 kB/s | 20 kB 00:00 2023-07-04T18:48:49.618 INFO:teuthology.orchestra.run.smithi082.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-07-04T18:48:50.043 INFO:teuthology.orchestra.run.smithi082.stdout:(93/131): ceph-test-18.0.0-4734.g207977f2.el8.x 9.1 MB/s | 48 MB 00:05 2023-07-04T18:48:50.102 INFO:teuthology.orchestra.run.smithi082.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 1.4 MB/s | 709 kB 00:00 2023-07-04T18:48:50.127 INFO:teuthology.orchestra.run.smithi082.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.3 MB/s | 198 kB 00:00 2023-07-04T18:48:50.169 INFO:teuthology.orchestra.run.smithi082.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-07-04T18:48:50.211 INFO:teuthology.orchestra.run.smithi082.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.4 MB/s | 354 kB 00:00 2023-07-04T18:48:50.311 INFO:teuthology.orchestra.run.smithi082.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 7.0 MB/s | 4.9 MB 00:00 2023-07-04T18:48:50.395 INFO:teuthology.orchestra.run.smithi082.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-04T18:48:50.438 INFO:teuthology.orchestra.run.smithi082.stdout:(100/131): postgresql-contrib-10.21-2.module+el 6.3 MB/s | 808 kB 00:00 2023-07-04T18:48:50.470 INFO:teuthology.orchestra.run.smithi082.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-07-04T18:48:50.512 INFO:teuthology.orchestra.run.smithi082.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-07-04T18:48:50.561 INFO:teuthology.orchestra.run.smithi082.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.3 MB/s | 200 kB 00:00 2023-07-04T18:48:50.579 INFO:teuthology.orchestra.run.smithi082.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 637 kB/s | 43 kB 00:00 2023-07-04T18:48:50.605 INFO:teuthology.orchestra.run.smithi082.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-07-04T18:48:50.622 INFO:teuthology.orchestra.run.smithi082.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-07-04T18:48:50.647 INFO:teuthology.orchestra.run.smithi082.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-04T18:48:50.664 INFO:teuthology.orchestra.run.smithi082.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-07-04T18:48:50.690 INFO:teuthology.orchestra.run.smithi082.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.5 MB/s | 109 kB 00:00 2023-07-04T18:48:50.715 INFO:teuthology.orchestra.run.smithi082.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-07-04T18:48:50.790 INFO:teuthology.orchestra.run.smithi082.stdout:(111/131): postgresql-server-10.21-2.module+el8 8.7 MB/s | 5.1 MB 00:00 2023-07-04T18:48:50.815 INFO:teuthology.orchestra.run.smithi082.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 186 kB/s | 19 kB 00:00 2023-07-04T18:48:50.841 INFO:teuthology.orchestra.run.smithi082.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-04T18:48:50.866 INFO:teuthology.orchestra.run.smithi082.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.1 MB/s | 560 kB 00:00 2023-07-04T18:48:50.916 INFO:teuthology.orchestra.run.smithi082.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 238 kB/s | 18 kB 00:00 2023-07-04T18:48:50.950 INFO:teuthology.orchestra.run.smithi082.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 4.1 MB/s | 559 kB 00:00 2023-07-04T18:48:50.983 INFO:teuthology.orchestra.run.smithi082.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-07-04T18:48:51.009 INFO:teuthology.orchestra.run.smithi082.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-07-04T18:48:51.050 INFO:teuthology.orchestra.run.smithi082.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.7 MB/s | 72 kB 00:00 2023-07-04T18:48:51.076 INFO:teuthology.orchestra.run.smithi082.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-07-04T18:48:51.101 INFO:teuthology.orchestra.run.smithi082.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-07-04T18:48:51.126 INFO:teuthology.orchestra.run.smithi082.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-04T18:48:51.177 INFO:teuthology.orchestra.run.smithi082.stdout:(123/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 10 MB/s | 2.3 MB 00:00 2023-07-04T18:48:51.212 INFO:teuthology.orchestra.run.smithi082.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 530 kB/s | 58 kB 00:00 2023-07-04T18:48:51.235 INFO:teuthology.orchestra.run.smithi082.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 354 kB/s | 39 kB 00:00 2023-07-04T18:48:51.261 INFO:teuthology.orchestra.run.smithi082.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 3.3 MB/s | 171 kB 00:00 2023-07-04T18:48:51.286 INFO:teuthology.orchestra.run.smithi082.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-07-04T18:48:51.319 INFO:teuthology.orchestra.run.smithi082.stdout:(128/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.4 MB/s | 644 kB 00:00 2023-07-04T18:48:51.370 INFO:teuthology.orchestra.run.smithi082.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-04T18:48:51.612 INFO:teuthology.orchestra.run.smithi082.stdout:(130/131): librbd1-18.0.0-4734.g207977f2.el8.x8 11 MB/s | 3.6 MB 00:00 2023-07-04T18:48:51.754 INFO:teuthology.orchestra.run.smithi082.stdout:(131/131): librados2-18.0.0-4734.g207977f2.el8. 7.4 MB/s | 3.7 MB 00:00 2023-07-04T18:48:51.756 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2023-07-04T18:48:51.757 INFO:teuthology.orchestra.run.smithi082.stdout:Total 23 MB/s | 205 MB 00:08 2023-07-04T18:48:52.599 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2023-07-04T18:48:52.719 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2023-07-04T18:48:52.719 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2023-07-04T18:48:53.828 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2023-07-04T18:48:53.831 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2023-07-04T18:48:58.691 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2023-07-04T18:48:58.866 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-04T18:48:59.023 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-04T18:48:59.275 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-04T18:48:59.443 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-04T18:48:59.542 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-04T18:48:59.743 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-04T18:49:00.147 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-04T18:49:00.359 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:49:00.762 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:49:00.976 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:49:01.088 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:49:01.259 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 8/134 2023-07-04T18:49:01.361 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 8/134 2023-07-04T18:49:01.568 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 9/134 2023-07-04T18:49:01.853 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-04T18:49:02.070 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-04T18:49:02.468 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-04T18:49:02.721 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-04T18:49:02.883 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 14/134 2023-07-04T18:49:02.992 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 14/134 2023-07-04T18:49:03.141 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 15/134 2023-07-04T18:49:03.229 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 15/134 2023-07-04T18:49:03.390 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-04T18:49:03.539 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-04T18:49:03.830 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-04T18:49:04.548 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-04T18:49:04.717 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-04T18:49:05.311 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-04T18:49:05.500 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-04T18:49:06.210 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-04T18:49:06.483 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-04T18:49:06.707 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-04T18:49:06.911 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-04T18:49:07.072 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-04T18:49:07.272 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-04T18:49:07.483 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-04T18:49:07.664 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:18.0.0-4734.g207977f2.el8. 26/134 2023-07-04T18:49:07.785 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:18.0.0-4734.g207977f2.el 27/134 2023-07-04T18:49:08.708 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_6 28/134 2023-07-04T18:49:09.012 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-04T18:49:09.292 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-04T18:49:10.134 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-04T18:49:10.357 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-04T18:49:12.114 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-04T18:49:12.433 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-04T18:49:12.668 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-04T18:49:12.850 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-04T18:49:13.020 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 36/134 2023-07-04T18:49:13.168 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:18.0.0-4734.g207977f2.el8.x86_6 37/134 2023-07-04T18:49:13.345 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-04T18:49:13.566 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-04T18:49:13.742 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-04T18:49:13.893 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-04T18:49:14.098 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-04T18:49:14.329 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-04T18:49:14.677 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-04T18:49:14.959 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-04T18:49:15.208 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-04T18:49:15.429 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-04T18:49:15.687 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-04T18:49:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-04T18:49:16.255 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-04T18:49:16.440 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-04T18:49:16.681 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-04T18:49:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-04T18:49:17.062 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-04T18:49:17.338 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-04T18:49:17.628 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-04T18:49:17.856 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-04T18:49:18.070 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-04T18:49:18.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-04T18:49:18.422 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-04T18:49:18.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-04T18:49:18.661 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-04T18:49:18.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-04T18:49:19.083 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-04T18:49:19.263 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-04T18:49:19.340 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-04T18:49:19.551 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-04T18:49:19.722 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-04T18:49:19.896 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-04T18:49:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-04T18:49:20.364 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-04T18:49:20.563 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-04T18:49:20.700 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-04T18:49:20.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-04T18:49:21.094 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-04T18:49:21.298 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-04T18:49:21.480 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-04T18:49:21.632 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-04T18:49:21.810 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-04T18:49:22.005 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-04T18:49:22.199 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-04T18:49:22.384 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-04T18:49:22.657 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-04T18:49:22.869 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-04T18:49:23.027 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-04T18:49:23.233 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-04T18:49:24.324 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-04T18:49:24.569 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-04T18:49:24.795 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-04T18:49:25.088 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-04T18:49:25.358 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-04T18:49:25.550 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-04T18:49:26.385 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-04T18:49:26.579 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 92/134 2023-07-04T18:49:26.649 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 92/134 2023-07-04T18:49:26.853 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 93/134 2023-07-04T18:49:30.540 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:49:30.753 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:49:31.356 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 94/134 2023-07-04T18:49:31.628 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 95/134 2023-07-04T18:49:31.720 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 95/134 2023-07-04T18:49:31.877 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 96/134 2023-07-04T18:49:50.603 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 96/134 2023-07-04T18:49:50.603 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2023-07-04T18:49:50.603 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /run 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2023-07-04T18:49:50.604 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2023-07-04T18:49:50.605 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /dev 2023-07-04T18:49:50.605 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:49:50.949 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 97/134 2023-07-04T18:49:52.517 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 97/134 2023-07-04T18:49:52.731 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 98/134 2023-07-04T18:49:52.949 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 98/134 2023-07-04T18:49:53.122 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 99/134 2023-07-04T18:49:53.577 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 99/134 2023-07-04T18:49:53.792 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 100/134 2023-07-04T18:49:53.806 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 100/134 2023-07-04T18:49:55.441 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:18.0.0-4734.g207977f2.el8.noarch 101/134 2023-07-04T18:49:55.608 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:18.0.0-4734.g207977f2.el8.noarch 101/134 2023-07-04T18:49:55.745 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.e 102/134 2023-07-04T18:49:55.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4734.g207977f2. 103/134 2023-07-04T18:49:56.121 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 104/134 2023-07-04T18:49:56.336 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 104/134 2023-07-04T18:49:56.639 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 105/134 2023-07-04T18:49:57.596 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 105/134 2023-07-04T18:49:57.746 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 106/134 2023-07-04T18:49:57.836 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 106/134 2023-07-04T18:49:58.183 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el 107/134 2023-07-04T18:49:58.389 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 108/134 2023-07-04T18:49:58.423 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 108/134 2023-07-04T18:49:58.597 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 109/134 2023-07-04T18:49:58.608 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 109/134 2023-07-04T18:49:59.923 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:18.0.0-4734.g207977f2.el8.x86_64 110/134 2023-07-04T18:50:00.164 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 111/134 2023-07-04T18:50:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 111/134 2023-07-04T18:50:00.354 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4734.g20797 112/134 2023-07-04T18:50:00.396 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4734.g20797 112/134 2023-07-04T18:50:00.554 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 113/134 2023-07-04T18:50:00.899 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 113/134 2023-07-04T18:50:01.054 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 114/134 2023-07-04T18:50:05.557 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 114/134 2023-07-04T18:50:05.829 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 115/134 2023-07-04T18:50:06.002 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-04T18:50:06.195 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-04T18:50:07.070 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-04T18:50:07.392 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-04T18:50:07.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 119/134 2023-07-04T18:50:07.798 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-04T18:50:07.965 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_ 121/134 2023-07-04T18:50:08.112 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-04T18:50:10.272 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:50:10.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:50:10.528 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-04T18:50:10.693 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 124/134 2023-07-04T18:50:11.018 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 125/134 2023-07-04T18:50:11.303 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 126/134 2023-07-04T18:50:11.637 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-04T18:50:11.835 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-04T18:50:11.844 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-04T18:50:12.026 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-04T18:50:12.201 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-04T18:50:12.288 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-04T18:50:12.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-04T18:50:12.524 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-04T18:50:12.524 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:50:12.755 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:50:12.755 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-04T18:50:13.006 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-04T18:50:13.209 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:50:13.721 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:50:13.721 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:18.0.0-4734.g207977f2.el8.x86_64 1/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 2/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 3/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 4/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4734.g20797 5/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 6/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 7/134 2023-07-04T18:50:13.722 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 8/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 9/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 10/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 11/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 12/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 13/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_ 14/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 15/134 2023-07-04T18:50:13.723 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 16/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:18.0.0-4734.g207977f2.el8.x86_6 17/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86 18/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 19/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4734.g207977f2.el 20/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:18.0.0-4734.g207977f2.el8. 21/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_6 22/134 2023-07-04T18:50:13.724 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 23/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 24/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 25/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 26/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 27/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 28/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4734.g207977f2. 29/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noa 30/134 2023-07-04T18:50:13.725 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.n 31/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4734.g207 32/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el 33/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 34/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.e 35/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 36/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:18.0.0-4734.g207977f2.el8.noarch 37/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 38/134 2023-07-04T18:50:13.726 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-04T18:50:13.727 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-04T18:50:13.728 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-04T18:50:13.729 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-04T18:50:13.730 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-04T18:50:13.731 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-04T18:50:13.732 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-04T18:50:13.733 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-04T18:50:13.734 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-04T18:50:13.735 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-04T18:50:13.736 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-04T18:50:13.737 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-04T18:50:13.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:18.0.0-4734.g207977f2.el8.x86_64 129/134 2023-07-04T18:50:13.739 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-04T18:50:13.739 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 131/134 2023-07-04T18:50:13.739 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-04T18:50:13.739 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-04T18:50:16.243 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-04T18:50:16.243 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:46 ago on Tue 04 Jul 2023 06:48:30 PM UTC. 2023-07-04T18:50:18.867 INFO:teuthology.orchestra.run.smithi082.stdout:Installed products updated. 2023-07-04T18:50:19.111 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:19.112 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2023-07-04T18:50:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-04T18:50:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.112 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: automake-1.16.1-7.el8.noarch 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: bison-3.0.4-10.el8.x86_64 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.113 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.114 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.115 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top-2:18.0.0-4734.g207977f2.el8.noarch 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: flex-2.6.1-9.el8.x86_64 2023-07-04T18:50:19.116 INFO:teuthology.orchestra.run.smithi082.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-04T18:50:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-04T18:50:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-3.el8.x86_64 2023-07-04T18:50:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-04T18:50:19.117 INFO:teuthology.orchestra.run.smithi082.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-04T18:50:19.118 INFO:teuthology.orchestra.run.smithi082.stdout: libpq-13.5-1.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.119 INFO:teuthology.orchestra.run.smithi082.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-04T18:50:19.120 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-04T18:50:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-04T18:50:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-04T18:50:19.121 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-04T18:50:19.122 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-04T18:50:19.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-04T18:50:19.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-2.6-1.el8.noarch 2023-07-04T18:50:19.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.127 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-04T18:50:19.128 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-04T18:50:19.129 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:18.0.0-4734.g207977f2.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-04T18:50:19.130 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-04T18:50:19.131 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-04T18:50:19.131 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-04T18:50:19.131 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:19.131 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2023-07-04T18:50:19.350 DEBUG:teuthology.parallel:result is None 2023-07-04T18:50:19.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=207977f2e527a6f2822da19c25b874c901ff76c8 2023-07-04T18:50:19.501 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-04T18:50:19.531 INFO:teuthology.orchestra.run.smithi077.stdout:18.0.0-4734.g207977f2.el8 2023-07-04T18:50:19.532 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4734.g207977f2.el8 2023-07-04T18:50:19.532 INFO:teuthology.task.install:The correct ceph version 18.0.0-4734.g207977f2 is installed. 2023-07-04T18:50:19.533 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=207977f2e527a6f2822da19c25b874c901ff76c8 2023-07-04T18:50:19.752 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-04T18:50:19.775 INFO:teuthology.orchestra.run.smithi082.stdout:18.0.0-4734.g207977f2.el8 2023-07-04T18:50:19.775 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4734.g207977f2.el8 2023-07-04T18:50:19.776 INFO:teuthology.task.install:The correct ceph version 18.0.0-4734.g207977f2 is installed. 2023-07-04T18:50:19.776 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-04T18:50:19.776 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:19.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-04T18:50:19.826 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:19.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-04T18:50:19.866 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-04T18:50:19.866 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:19.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2023-07-04T18:50:19.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-04T18:50:19.965 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:19.965 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2023-07-04T18:50:19.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-04T18:50:20.059 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-04T18:50:20.060 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:20.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-04T18:50:20.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-04T18:50:20.158 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:20.158 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-04T18:50:20.185 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-04T18:50:20.253 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-04T18:50:20.254 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:20.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2023-07-04T18:50:20.283 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-04T18:50:20.351 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:20.352 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2023-07-04T18:50:20.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-04T18:50:20.443 INFO:teuthology.run_tasks:Running task ceph... 2023-07-04T18:50:20.520 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-07-04T18:50:20.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2023-07-04T18:50:20.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2023-07-04T18:50:20.550 INFO:tasks.ceph:Disabling ceph logrotate... 2023-07-04T18:50:20.550 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-04T18:50:20.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-04T18:50:20.618 INFO:tasks.ceph:Creating extra log directories... 2023-07-04T18:50:20.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-04T18:50:20.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-04T18:50:20.694 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-07-04T18:50:20.694 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-07-04T18:50:20.695 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-07-04T18:50:20.695 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:20.695 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-07-04T18:50:20.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:20.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:20.916 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:20.917 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-07-04T18:50:20.933 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:21.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:21.099 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-07-04T18:50:21.099 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-07-04T18:50:21.100 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi/7326492', 'branch': 'wip-vshankar-testing-20230704.040242', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7326492', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '84139c18d56f518615369e8310ae4ecf07a36dcd'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-07-04_11:45:30-fs-wip-vshankar-testing-20230704.040242-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20230704.040242'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20230704.040242', 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20230704.040242', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_207977f2e527a6f2822da19c25b874c901ff76c8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '207977f2e527a6f2822da19c25b874c901ff76c8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi077.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi082.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '84139c18d56f518615369e8310ae4ecf07a36dcd'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '207977f2e527a6f2822da19c25b874c901ff76c8'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.tests_from_xfstests_dev']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b53f236c1077f1d295baafb9e0ca9a4b447019d7', 'timestamp': '2023-07-04_11:45:30', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-07-04T18:50:21.100 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-04T18:50:21.103 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-04T18:50:21.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2023-07-04T18:50:21.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2023-07-04T18:50:21.217 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:21.217 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2023-07-04T18:50:21.241 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-04T18:50:21.241 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2023-07-04T18:50:21.298 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 378446 Links: 1 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-07-04 18:47:13.507573678 +0000 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-07-04 18:43:07.679516568 +0000 2023-07-04T18:50:21.299 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-07-04 18:43:07.679516568 +0000 2023-07-04T18:50:21.300 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-07-04T18:50:21.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-04T18:50:21.363 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-07-04T18:50:21.363 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-07-04T18:50:21.364 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000250396 s, 2.0 MB/s 2023-07-04T18:50:21.365 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-04T18:50:21.423 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2023-07-04T18:50:21.480 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-04T18:50:21.480 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:21.480 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 378607 Links: 1 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-07-04 18:47:13.507573678 +0000 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-07-04 18:43:08.198505233 +0000 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-07-04 18:43:08.198505233 +0000 2023-07-04T18:50:21.481 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-07-04T18:50:21.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-04T18:50:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-07-04T18:50:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-07-04T18:50:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00022468 s, 2.3 MB/s 2023-07-04T18:50:21.546 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-04T18:50:21.605 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2023-07-04T18:50:21.662 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-04T18:50:21.662 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:21.662 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 381176 Links: 1 2023-07-04T18:50:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-07-04 18:47:13.507573678 +0000 2023-07-04T18:50:21.663 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-07-04 18:43:08.710494051 +0000 2023-07-04T18:50:21.664 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-07-04 18:43:08.710494051 +0000 2023-07-04T18:50:21.664 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-07-04T18:50:21.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-04T18:50:21.727 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-07-04T18:50:21.728 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-07-04T18:50:21.728 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000228577 s, 2.2 MB/s 2023-07-04T18:50:21.729 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-04T18:50:21.787 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2023-07-04T18:50:21.846 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-04T18:50:21.846 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:21.846 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 378679 Links: 1 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-07-04 18:47:13.507573678 +0000 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-07-04 18:43:09.244482388 +0000 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-07-04 18:43:09.244482388 +0000 2023-07-04T18:50:21.847 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-07-04T18:50:21.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-04T18:50:21.910 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-07-04T18:50:21.911 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-07-04T18:50:21.911 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000194818 s, 2.6 MB/s 2023-07-04T18:50:21.912 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-04T18:50:21.970 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:21.971 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:21.972 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2023-07-04T18:50:21.991 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-04T18:50:21.991 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2023-07-04T18:50:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-04T18:50:22.051 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:22.051 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 371433 Links: 1 2023-07-04T18:50:22.051 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:22.051 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:22.052 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-07-04 18:50:13.414969415 +0000 2023-07-04T18:50:22.052 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-07-04 18:43:19.293307396 +0000 2023-07-04T18:50:22.052 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-07-04 18:43:19.293307396 +0000 2023-07-04T18:50:22.052 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-07-04T18:50:22.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-04T18:50:22.120 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-07-04T18:50:22.120 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-07-04T18:50:22.121 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000268099 s, 1.9 MB/s 2023-07-04T18:50:22.121 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-04T18:50:22.179 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2023-07-04T18:50:22.235 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-04T18:50:22.236 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:22.236 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 372558 Links: 1 2023-07-04T18:50:22.237 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:22.237 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:22.237 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-07-04 18:50:13.414969415 +0000 2023-07-04T18:50:22.237 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-07-04 18:43:19.814306957 +0000 2023-07-04T18:50:22.238 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-07-04 18:43:19.814306957 +0000 2023-07-04T18:50:22.238 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-07-04T18:50:22.238 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-04T18:50:22.300 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-07-04T18:50:22.300 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-07-04T18:50:22.300 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000240934 s, 2.1 MB/s 2023-07-04T18:50:22.301 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-04T18:50:22.359 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2023-07-04T18:50:22.417 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-04T18:50:22.417 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 380102 Links: 1 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-07-04 18:50:13.414969415 +0000 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-07-04 18:43:20.333306520 +0000 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-07-04 18:43:20.333306520 +0000 2023-07-04T18:50:22.418 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-07-04T18:50:22.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-04T18:50:22.482 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-07-04T18:50:22.482 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-07-04T18:50:22.482 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000206476 s, 2.5 MB/s 2023-07-04T18:50:22.483 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-04T18:50:22.542 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2023-07-04T18:50:22.599 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-04T18:50:22.599 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-04T18:50:22.599 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 6h/6d Inode: 371613 Links: 1 2023-07-04T18:50:22.599 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-04T18:50:22.600 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2023-07-04T18:50:22.600 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2023-07-04 18:50:13.414969415 +0000 2023-07-04T18:50:22.600 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2023-07-04 18:43:20.853306082 +0000 2023-07-04T18:50:22.600 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2023-07-04 18:43:20.853306082 +0000 2023-07-04T18:50:22.600 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2023-07-04T18:50:22.601 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-04T18:50:22.665 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2023-07-04T18:50:22.666 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2023-07-04T18:50:22.666 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000205701 s, 2.5 MB/s 2023-07-04T18:50:22.667 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-04T18:50:22.725 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:22.726 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-07-04T18:50:22.726 INFO:tasks.ceph:Generating config... 2023-07-04T18:50:22.728 INFO:tasks.ceph:[client] client mount timeout = 600 2023-07-04T18:50:22.729 INFO:tasks.ceph:[client] debug client = 20 2023-07-04T18:50:22.729 INFO:tasks.ceph:[client] debug ms = 1 2023-07-04T18:50:22.729 INFO:tasks.ceph:[client] pid file = 2023-07-04T18:50:22.729 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-07-04T18:50:22.729 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-07-04T18:50:22.730 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-07-04T18:50:22.730 INFO:tasks.ceph:[mds] debug mds = 20 2023-07-04T18:50:22.730 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-07-04T18:50:22.730 INFO:tasks.ceph:[mds] debug ms = 1 2023-07-04T18:50:22.730 INFO:tasks.ceph:[mds] mds debug frag = True 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-07-04T18:50:22.731 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-07-04T18:50:22.732 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-07-04T18:50:22.732 INFO:tasks.ceph:[mgr] debug ms = 1 2023-07-04T18:50:22.732 INFO:tasks.ceph:[mon] debug mon = 20 2023-07-04T18:50:22.732 INFO:tasks.ceph:[mon] debug ms = 1 2023-07-04T18:50:22.732 INFO:tasks.ceph:[mon] debug paxos = 20 2023-07-04T18:50:22.733 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-07-04T18:50:22.733 INFO:tasks.ceph:[osd] bdev async discard = True 2023-07-04T18:50:22.733 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-07-04T18:50:22.733 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-07-04T18:50:22.733 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] debug ms = 1 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] debug osd = 20 2023-07-04T18:50:22.734 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-07-04T18:50:22.735 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-07-04T18:50:22.735 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-07-04T18:50:22.735 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-07-04T18:50:22.735 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-07-04T18:50:22.735 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-07-04T18:50:22.736 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-07-04T18:50:22.736 INFO:tasks.ceph:Setting up mon.a... 2023-07-04T18:50:22.736 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-07-04T18:50:22.798 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2023-07-04T18:50:22.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-07-04T18:50:22.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-04T18:50:22.875 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77'), ('mon.b', '172.21.15.82'), ('mon.c', '[v2:172.21.15.82:3301,v1:172.21.15.82:6790]')] 2023-07-04T18:50:22.876 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77,172.21.15.82,[v2:172.21.15.82:3301,v1:172.21.15.82:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-07-04T18:50:22.877 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:22.877 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-07-04T18:50:22.932 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --add b 172.21.15.82 --addv c '[v2:172.21.15.82:3301,v1:172.21.15.82:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:23.019 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:23.020 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5 2023-07-04T18:50:23.020 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = pacific 2023-07-04T18:50:23.020 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2023-07-04T18:50:23.020 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2023-07-04T18:50:23.019286+0000 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:created 2023-07-04T18:50:23.019286+0000 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 16 (pacific) 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2023-07-04T18:50:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:2: [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] mon.c 2023-07-04T18:50:23.022 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-07-04T18:50:23.023 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-07-04T18:50:23.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5... 2023-07-04T18:50:23.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-04T18:50:23.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-04T18:50:23.125 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2023-07-04T18:50:23.125 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2023-07-04T18:50:23.125 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-04T18:50:23.125 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2023-07-04T18:50:23.126 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2023-07-04T18:50:23.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.128 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2023-07-04T18:50:23.128 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-04T18:50:23.128 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.128 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2023-07-04T18:50:23.128 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.129 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2023-07-04T18:50:23.129 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2023-07-04T18:50:23.129 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2023-07-04T18:50:23.129 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2023-07-04T18:50:23.130 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2023-07-04T18:50:23.130 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2023-07-04T18:50:23.130 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2023-07-04T18:50:23.131 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-04T18:50:23.131 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-04T18:50:23.131 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2023-07-04T18:50:23.132 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.132 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-04T18:50:23.132 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2023-07-04T18:50:23.133 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.82,[v2:172.21.15.82:3301,v1:172.21.15.82:6790]" 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on skipped message = False 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.134 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2023-07-04T18:50:23.135 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2023-07-04T18:50:23.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.135 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2023-07-04T18:50:23.135 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2023-07-04T18:50:23.135 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2023-07-04T18:50:23.136 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.136 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.137 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.138 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2023-07-04T18:50:23.139 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-07-04T18:50:23.140 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2023-07-04T18:50:23.141 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-07-04T18:50:23.142 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-07-04T18:50:23.143 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.144 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.145 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2023-07-04T18:50:23.146 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-07-04T18:50:23.147 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = "" 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2023-07-04T18:50:23.148 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds balancer = 20 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2023-07-04T18:50:23.149 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2023-07-04T18:50:23.150 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2023-07-04T18:50:23.150 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2023-07-04T18:50:23.150 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2023-07-04T18:50:23.150 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2023-07-04T18:50:23.150 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2023-07-04T18:50:23.152 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2023-07-04T18:50:23.152 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2023-07-04T18:50:23.152 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-04T18:50:23.152 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2023-07-04T18:50:23.152 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2023-07-04T18:50:23.153 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2023-07-04T18:50:23.154 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2023-07-04T18:50:23.155 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.156 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2023-07-04T18:50:23.157 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "172.21.15.77,172.21.15.82,[v2:172.21.15.82:3301,v1:172.21.15.82:6790]" 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on skipped message = False 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2023-07-04T18:50:23.158 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2023-07-04T18:50:23.159 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.160 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2023-07-04T18:50:23.160 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2023-07-04T18:50:23.160 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2023-07-04T18:50:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.161 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.162 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2023-07-04T18:50:23.163 INFO:teuthology.orchestra.run.smithi082.stdout: bdev async discard = True 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: bdev enable discard = True 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore allocator = bitmap 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 1/20 2023-07-04T18:50:23.164 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 1/20 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 4/10 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2023-07-04T18:50:23.165 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2023-07-04T18:50:23.166 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2023-07-04T18:50:23.166 INFO:teuthology.orchestra.run.smithi082.stdout: osd op complaint time = 180 2023-07-04T18:50:23.166 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.166 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2023-07-04T18:50:23.166 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2023-07-04T18:50:23.167 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2023-07-04T18:50:23.168 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2023-07-04T18:50:23.169 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2023-07-04T18:50:23.170 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: mon op complaint time = 120 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2023-07-04T18:50:23.171 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2023-07-04T18:50:23.172 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-04T18:50:23.172 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-04T18:50:23.172 INFO:teuthology.orchestra.run.smithi082.stdout: client mount timeout = 600 2023-07-04T18:50:23.172 INFO:teuthology.orchestra.run.smithi082.stdout: debug client = 20 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = "" 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout: rados mon op timeout = 900 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout: rados osd op timeout = 900 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2023-07-04T18:50:23.173 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.a] 2023-07-04T18:50:23.174 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.c] 2023-07-04T18:50:23.174 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.b] 2023-07-04T18:50:23.174 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.d] 2023-07-04T18:50:23.174 INFO:teuthology.orchestra.run.smithi082.stdout:[mds] 2023-07-04T18:50:23.174 INFO:teuthology.orchestra.run.smithi082.stdout: debug mds = 20 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: debug mds balancer = 20 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: mds debug frag = True 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: mds debug scatterstat = True 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: mds op complaint time = 180 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: mds verify scatter = True 2023-07-04T18:50:23.175 INFO:teuthology.orchestra.run.smithi082.stdout: osd op complaint time = 180 2023-07-04T18:50:23.176 INFO:teuthology.orchestra.run.smithi082.stdout: rados mon op timeout = 900 2023-07-04T18:50:23.176 INFO:teuthology.orchestra.run.smithi082.stdout: rados osd op timeout = 900 2023-07-04T18:50:23.176 INFO:tasks.ceph:Creating admin key on mon.a... 2023-07-04T18:50:23.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-07-04T18:50:23.249 INFO:tasks.ceph:Copying monmap to all nodes... 2023-07-04T18:50:23.250 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:23.251 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-07-04T18:50:23.268 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:23.269 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-07-04T18:50:23.325 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:50:23.325 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:23.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-04T18:50:23.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-04T18:50:23.401 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:23.402 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:23.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:50:23.459 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:23.460 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-04T18:50:23.460 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-04T18:50:23.497 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:23.498 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:23.555 INFO:tasks.ceph:Setting up mon nodes... 2023-07-04T18:50:23.556 INFO:tasks.ceph:Setting up mgr nodes... 2023-07-04T18:50:23.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-07-04T18:50:23.610 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-07-04T18:50:23.612 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-07-04T18:50:23.678 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-07-04T18:50:23.680 INFO:tasks.ceph:Setting up mds nodes... 2023-07-04T18:50:23.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-07-04T18:50:23.734 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-07-04T18:50:23.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-07-04T18:50:23.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-07-04T18:50:23.892 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-07-04T18:50:23.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-07-04T18:50:23.922 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-07-04T18:50:23.978 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-07-04T18:50:23.981 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-07-04T18:50:24.010 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-07-04T18:50:24.117 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-07-04T18:50:24.117 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-07-04T18:50:24.145 INFO:tasks.ceph_client:Setting up client nodes... 2023-07-04T18:50:24.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-04T18:50:24.191 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-07-04T18:50:24.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-07-04T18:50:24.285 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-07-04T18:50:24.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-07-04T18:50:24.341 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-07-04T18:50:24.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-07-04T18:50:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-07-04T18:50:24.412 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-07-04T18:50:24.413 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-07-04T18:50:24.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-07-04T18:50:24.439 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:24.440 INFO:tasks.ceph:role: osd.0 2023-07-04T18:50:24.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:50:24.441 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-04T18:50:24.515 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:24.515 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:24.515 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:24.516 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:24.516 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:24.516 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:24.516 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:24.517 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:24.517 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:24.947 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-07-04T18:50:24.948 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-07-04T18:50:24.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-07-04T18:50:25.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-07-04T18:50:25.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-07-04T18:50:25.233 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:25.233 INFO:tasks.ceph:role: osd.1 2023-07-04T18:50:25.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:50:25.234 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-04T18:50:25.306 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:25.307 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:25.307 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:25.307 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:25.307 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:25.307 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:25.308 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:25.308 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:25.308 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:25.308 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:25.731 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-07-04T18:50:25.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-07-04T18:50:25.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-07-04T18:50:25.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-07-04T18:50:25.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-07-04T18:50:25.950 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:25.951 INFO:tasks.ceph:role: osd.2 2023-07-04T18:50:25.951 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:50:25.951 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-04T18:50:26.024 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:26.024 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:26.025 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-07-04T18:50:26.474 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-07-04T18:50:26.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-07-04T18:50:26.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-07-04T18:50:26.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-07-04T18:50:26.723 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:26.723 INFO:tasks.ceph:role: osd.3 2023-07-04T18:50:26.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:50:26.724 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-07-04T18:50:26.797 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:26.797 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:26.797 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:26.798 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:26.799 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:27.235 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-07-04T18:50:27.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-07-04T18:50:27.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-07-04T18:50:27.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-07-04T18:50:27.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:27.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.562+0000 7fc347df7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-07-04T18:50:27.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.563+0000 7fc347df7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-07-04T18:50:27.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.563+0000 7fc347df7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-04T18:50:27.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.563+0000 7fc347df7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-04T18:50:27.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.563+0000 7fc347df7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-04T18:50:27.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:27.563+0000 7fc347df7640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-07-04T18:50:30.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-04T18:50:30.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:30.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.550+0000 7f6385ce0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-07-04T18:50:30.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.550+0000 7f6385ce0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-07-04T18:50:30.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.550+0000 7f6385ce0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-04T18:50:30.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.550+0000 7f6385ce0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-04T18:50:30.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.550+0000 7f6385ce0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-04T18:50:30.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:30.551+0000 7f6385ce0640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-07-04T18:50:33.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-04T18:50:33.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:33.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-07-04T18:50:33.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-07-04T18:50:33.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-04T18:50:33.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-04T18:50:33.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-04T18:50:33.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:33.537+0000 7faf962f4640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-07-04T18:50:36.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-04T18:50:36.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:36.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.528+0000 7f490fe32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-07-04T18:50:36.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.528+0000 7f490fe32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-07-04T18:50:36.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.528+0000 7f490fe32640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-04T18:50:36.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.528+0000 7f490fe32640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-04T18:50:36.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.528+0000 7f490fe32640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-04T18:50:36.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:36.529+0000 7f490fe32640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-07-04T18:50:39.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-04T18:50:39.423 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-07-04T18:50:39.454 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:39.454 INFO:tasks.ceph:role: osd.4 2023-07-04T18:50:39.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:50:39.455 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-04T18:50:39.532 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:39.533 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:39.534 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:39.534 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:39.534 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:39.958 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-07-04T18:50:39.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-07-04T18:50:39.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-07-04T18:50:40.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-07-04T18:50:40.181 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-07-04T18:50:40.303 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:40.303 INFO:tasks.ceph:role: osd.5 2023-07-04T18:50:40.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:50:40.303 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-04T18:50:40.339 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:40.339 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:40.340 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:40.341 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:40.341 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:40.756 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-07-04T18:50:40.758 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-07-04T18:50:40.758 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-07-04T18:50:40.872 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-07-04T18:50:40.904 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-07-04T18:50:40.972 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:40.973 INFO:tasks.ceph:role: osd.6 2023-07-04T18:50:40.973 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:50:40.973 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-04T18:50:41.049 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:41.050 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:41.051 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:41.490 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-07-04T18:50:41.492 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-07-04T18:50:41.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-07-04T18:50:41.638 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-07-04T18:50:41.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-07-04T18:50:41.737 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-07-04T18:50:41.737 INFO:tasks.ceph:role: osd.7 2023-07-04T18:50:41.737 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:50:41.737 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-07-04T18:50:41.811 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-07-04T18:50:41.811 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-04T18:50:41.812 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-04T18:50:41.813 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-04T18:50:42.261 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2023-07-04T18:50:42.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2023-07-04T18:50:42.264 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-07-04T18:50:42.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-07-04T18:50:42.475 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.582+0000 7f6344784640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-07-04T18:50:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.582+0000 7f6344784640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-07-04T18:50:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.582+0000 7f6344784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-04T18:50:42.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.582+0000 7f6344784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-04T18:50:42.585 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.582+0000 7f6344784640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-04T18:50:42.585 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:42.583+0000 7f6344784640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-07-04T18:50:45.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-04T18:50:45.486 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:45.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.575+0000 7f8bc1900640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-07-04T18:50:45.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.575+0000 7f8bc1900640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-07-04T18:50:45.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.575+0000 7f8bc1900640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-04T18:50:45.577 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.575+0000 7f8bc1900640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-04T18:50:45.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.575+0000 7f8bc1900640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-04T18:50:45.578 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:45.576+0000 7f8bc1900640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-07-04T18:50:48.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-04T18:50:48.468 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:48.559 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.557+0000 7f86e5ee2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-07-04T18:50:48.559 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.557+0000 7f86e5ee2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-07-04T18:50:48.560 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.557+0000 7f86e5ee2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-07-04T18:50:48.560 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.557+0000 7f86e5ee2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-07-04T18:50:48.560 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.557+0000 7f86e5ee2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-07-04T18:50:48.560 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:48.558+0000 7f86e5ee2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-07-04T18:50:51.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:51.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:50:51.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-07-04T18:50:51.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:51.542 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.540+0000 7f5c6e680640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-07-04T18:50:51.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.541+0000 7f5c6e680640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-07-04T18:50:51.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.541+0000 7f5c6e680640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-07-04T18:50:51.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.541+0000 7f5c6e680640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-07-04T18:50:51.543 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.541+0000 7f5c6e680640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-07-04T18:50:51.544 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:51.541+0000 7f5c6e680640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-07-04T18:50:54.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-07-04T18:50:54.470 INFO:tasks.ceph:Reading keys from all nodes... 2023-07-04T18:50:54.471 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-07-04T18:50:54.502 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-07-04T18:50:54.574 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.574 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-07-04T18:50:54.645 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.646 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-07-04T18:50:54.715 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-07-04T18:50:54.784 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-07-04T18:50:54.851 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:54.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-07-04T18:50:54.919 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:54.920 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-07-04T18:50:54.953 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:54.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-07-04T18:50:55.022 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-07-04T18:50:55.091 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.091 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-07-04T18:50:55.160 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-07-04T18:50:55.226 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.227 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-07-04T18:50:55.293 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.294 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-07-04T18:50:55.360 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:55.360 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-07-04T18:50:55.380 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:55.380 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-07-04T18:50:55.442 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.442 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-07-04T18:50:55.460 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:55.460 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-07-04T18:50:55.516 INFO:tasks.ceph:Adding keys to all mons... 2023-07-04T18:50:55.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2023-07-04T18:50:55.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2023-07-04T18:50:55.559 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2023-07-04T18:50:55.559 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRkI/tdJBAAcEwV7OEFu5QIKthn3bPbuw== 2023-07-04T18:50:55.559 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRky+nFKxAAezOJXbPY3CsDRiEcGgACTA== 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRkAH0oNRAAAZUQHvxEfVyCKzH6njiD5g== 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDzaaRk61edIRAAdAERV/JhvfaF//RGvGuOHg== 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2023-07-04T18:50:55.560 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD2aaRk6fnhIBAAf8KlJRpZqp7VVp5MFYmX0Q== 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD5aaRkqKUWIBAAjKLo1hLm/xz62mRUBZB8KQ== 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD8aaRkSkuPHxAApozzYyTZeymH06LWVdfDGA== 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRkY0JoKBAAsA6b/+K5BpY9FyOrOjJgFQ== 2023-07-04T18:50:55.561 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRky/VMOhAAXB5gnz520M0KrkGtvr1MAw== 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRkaJZLBhAA3CaTJFrHF9QmApJPgMDtvw== 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQACaqRk7l7KIhAAdYdujaxh0QN9VlQ8xzb/ZA== 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAFaqRku2JfIhAATRwR47BbceGZqHKEdF5Tmw== 2023-07-04T18:50:55.562 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAIaqRkLmVMIRAATAKAgx2ikSIIA/gJdz7Pvw== 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQALaqRkmJlRIBAAiBPJBA2xrcidHRyPI7zYvA== 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRkpsg9CxAA3I+2BiR3flfH2NGjwz9V9Q== 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout:[client.2] 2023-07-04T18:50:55.563 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRkZ1v8EBAAzkcz24OS5W6SAApn3aR7JA== 2023-07-04T18:50:55.564 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2023-07-04T18:50:55.564 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRk+q1aFBAAkptDNcBtzvvWqbPBOunKlA== 2023-07-04T18:50:55.564 INFO:teuthology.orchestra.run.smithi077.stdout:[client.3] 2023-07-04T18:50:55.564 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRknsfHFxAAMNKVa7AqjR4vqoBVuVDQVw== 2023-07-04T18:50:55.580 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.y] 2023-07-04T18:50:55.581 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvaaRkI/tdJBAAcEwV7OEFu5QIKthn3bPbuw== 2023-07-04T18:50:55.581 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.a] 2023-07-04T18:50:55.581 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvaaRky+nFKxAAezOJXbPY3CsDRiEcGgACTA== 2023-07-04T18:50:55.581 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.c] 2023-07-04T18:50:55.581 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvaaRkAH0oNRAAAZUQHvxEfVyCKzH6njiD5g== 2023-07-04T18:50:55.582 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2023-07-04T18:50:55.582 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDzaaRk61edIRAAdAERV/JhvfaF//RGvGuOHg== 2023-07-04T18:50:55.582 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2023-07-04T18:50:55.582 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQD2aaRk6fnhIBAAf8KlJRpZqp7VVp5MFYmX0Q== 2023-07-04T18:50:55.582 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQD5aaRkqKUWIBAAjKLo1hLm/xz62mRUBZB8KQ== 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQD8aaRkSkuPHxAApozzYyTZeymH06LWVdfDGA== 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvaaRkY0JoKBAAsA6b/+K5BpY9FyOrOjJgFQ== 2023-07-04T18:50:55.583 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.b] 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDvaaRky/VMOhAAXB5gnz520M0KrkGtvr1MAw== 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout:[mds.d] 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDwaaRkaJZLBhAA3CaTJFrHF9QmApJPgMDtvw== 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQACaqRk7l7KIhAAdYdujaxh0QN9VlQ8xzb/ZA== 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2023-07-04T18:50:55.584 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAFaqRku2JfIhAATRwR47BbceGZqHKEdF5Tmw== 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.6] 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAIaqRkLmVMIRAATAKAgx2ikSIIA/gJdz7Pvw== 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.7] 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQALaqRkmJlRIBAAiBPJBA2xrcidHRyPI7zYvA== 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDwaaRkpsg9CxAA3I+2BiR3flfH2NGjwz9V9Q== 2023-07-04T18:50:55.585 INFO:teuthology.orchestra.run.smithi082.stdout:[client.2] 2023-07-04T18:50:55.586 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDwaaRkZ1v8EBAAzkcz24OS5W6SAApn3aR7JA== 2023-07-04T18:50:55.586 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2023-07-04T18:50:55.586 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDwaaRk+q1aFBAAkptDNcBtzvvWqbPBOunKlA== 2023-07-04T18:50:55.586 INFO:teuthology.orchestra.run.smithi082.stdout:[client.3] 2023-07-04T18:50:55.586 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQDwaaRknsfHFxAAMNKVa7AqjR4vqoBVuVDQVw== 2023-07-04T18:50:55.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-04T18:50:55.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-04T18:50:55.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:55.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:55.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:55.744 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:55.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.803 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:55.968 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-04T18:50:56.025 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-04T18:50:56.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:56.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:56.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:56.141 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-07-04T18:50:56.197 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.199 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.251 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.310 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-04T18:50:56.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-04T18:50:56.655 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-07-04T18:50:56.655 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-07-04T18:50:56.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-04T18:50:56.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-07-04T18:50:56.806 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-07-04T18:50:56.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-04T18:50:56.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-07-04T18:50:56.958 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-07-04T18:50:57.025 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-04T18:50:57.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-07-04T18:50:57.148 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:57.150 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-07-04T18:50:57.203 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-07-04T18:50:57.204 INFO:tasks.ceph.mon.a:Restarting daemon 2023-07-04T18:50:57.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-07-04T18:50:57.207 INFO:tasks.ceph.mon.a:Started 2023-07-04T18:50:57.208 INFO:tasks.ceph.mon.b:Restarting daemon 2023-07-04T18:50:57.208 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-07-04T18:50:57.245 INFO:tasks.ceph.mon.b:Started 2023-07-04T18:50:57.246 INFO:tasks.ceph.mon.c:Restarting daemon 2023-07-04T18:50:57.246 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-07-04T18:50:57.248 INFO:tasks.ceph.mon.c:Started 2023-07-04T18:50:57.248 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-07-04T18:50:57.248 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-07-04T18:50:57.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-07-04T18:50:57.251 INFO:tasks.ceph.mgr.y:Started 2023-07-04T18:50:57.251 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-07-04T18:50:57.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-07-04T18:50:57.253 INFO:tasks.ceph.mgr.x:Started 2023-07-04T18:50:57.253 INFO:tasks.ceph:Setting crush tunables to default 2023-07-04T18:50:57.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2023-07-04T18:50:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.374+0000 7ff619039700 1 Processor -- start 2023-07-04T18:50:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.374+0000 7ff619039700 1 -- start start 2023-07-04T18:50:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff619039700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 0x7ff61406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff619039700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140b3f60 con 0x7ff6141590b0 2023-07-04T18:50:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff619039700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6140b4b00 con 0x7ff6140b3330 2023-07-04T18:50:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff619039700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61406caa0 con 0x7ff61406cfc0 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff612d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff6141590b0 0x7ff61415d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50398/0 (socket says 172.21.15.77:50398) 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff612d9d700 1 -- 172.21.15.77:0/613620561 learned_addr learned my addr 172.21.15.77:0/613620561 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 0x7ff61406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 msgr2=0x7ff61406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 msgr2=0x7ff61406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 0x7ff61406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T18:50:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.375+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 0x7ff61406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T18:50:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.569+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727476112 0 0) 0x7ff6140b3f60 con 0x7ff6141590b0 2023-07-04T18:50:57.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.569+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc0035b0 con 0x7ff6141590b0 2023-07-04T18:50:57.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646743240 0 0) 0x7ff6140b4b00 con 0x7ff6140b3330 2023-07-04T18:50:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6140b3f60 con 0x7ff6140b3330 2023-07-04T18:50:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566885118 0 0) 0x7ff6140b3f60 con 0x7ff6140b3330 2023-07-04T18:50:57.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 msgr2=0x7ff61406d3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 --2- 172.21.15.77:0/613620561 >> v2:172.21.15.77:3300/0 conn(0x7ff61406cfc0 0x7ff61406d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 >> v1:172.21.15.77:6789/0 conn(0x7ff6141590b0 legacy=0x7ff61415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61406d900 con 0x7ff6140b3330 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7ff608001670 con 0x7ff6140b3330 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 >> v1:172.21.15.82:6789/0 conn(0x7ff6140b3330 legacy=0x7ff6140b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5fc003b90 0x7ff5fc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5fc004850 0x7ff5fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff5fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61406caa0 con 0x7ff5fc005270 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc00c1b0 con 0x7ff5fc004850 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc00c370 con 0x7ff5fc003b90 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff5fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff612d9d700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5fc003b90 0x7ff5fc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61259c700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5fc004850 0x7ff5fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5fc003b90 msgr2=0x7ff5fc003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5fc003b90 0x7ff5fc003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5fc004850 msgr2=0x7ff5fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5fc004850 0x7ff5fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.570+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6140b3f60 con 0x7ff5fc005270 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff61359e700 1 -- 172.21.15.77:0/613620561 reap_dead start 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff61359e700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff5fc00ba10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff604009fd0 tx=0x7ff5fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709f529d955635cb server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:57.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604007e20 con 0x7ff5fc005270 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff604003a20 con 0x7ff5fc005270 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff611d9b700 1 -- 172.21.15.77:0/613620561 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604003dd0 con 0x7ff5fc005270 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff619039700 1 -- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 msgr2=0x7ff5fc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff619039700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff5fc00ba10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff604009fd0 tx=0x7ff5fc005880 comp rx=0 tx=0).stop 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff619039700 1 -- 172.21.15.77:0/613620561 shutdown_connections 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff619039700 1 --2- 172.21.15.77:0/613620561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff5fc00ba10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.571+0000 7ff619039700 1 -- 172.21.15.77:0/613620561 >> 172.21.15.77:0/613620561 conn(0x7ff61406af50 msgr2=0x7ff61406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:57.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.572+0000 7ff619039700 1 -- 172.21.15.77:0/613620561 shutdown_connections 2023-07-04T18:50:57.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.572+0000 7ff619039700 1 -- 172.21.15.77:0/613620561 wait complete. 2023-07-04T18:50:57.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.573+0000 7ff619039700 1 Processor -- start 2023-07-04T18:50:57.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.573+0000 7ff619039700 1 -- start start 2023-07-04T18:50:57.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff614158020 0x7ff61414df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff614069e40 0x7ff61406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:57.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61414cbd0 con 0x7ff5fc005270 2023-07-04T18:50:57.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61406a800 con 0x7ff614069e40 2023-07-04T18:50:57.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff619039700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61414a490 con 0x7ff614158020 2023-07-04T18:50:57.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff61359e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff614069e40 0x7ff61406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43280/0 (socket says 172.21.15.77:43280) 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 -- 172.21.15.77:0/2027861767 learned_addr learned my addr 172.21.15.77:0/2027861767 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff61259c700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff614158020 0x7ff61414df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 -- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff614158020 msgr2=0x7ff61414df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff614158020 0x7ff61414df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 -- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff614069e40 msgr2=0x7ff61406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.574+0000 7ff612d9d700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff614069e40 0x7ff61406a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff612d9d700 1 -- 172.21.15.77:0/2027861767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614157b90 con 0x7ff5fc005270 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff612d9d700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff60c004f90 tx=0x7ff60c007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c012560 con 0x7ff5fc005270 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c012700 con 0x7ff5fc005270 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c012ab0 con 0x7ff5fc005270 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff614156f80 con 0x7ff5fc005270 2023-07-04T18:50:57.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff614157540 con 0x7ff5fc005270 2023-07-04T18:50:57.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff60c01d460 con 0x7ff5fc005270 2023-07-04T18:50:57.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.575+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff60c013560 con 0x7ff5fc005270 2023-07-04T18:50:57.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.576+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61406d6f0 con 0x7ff5fc005270 2023-07-04T18:50:57.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.578+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7ff60c01d8f0 con 0x7ff5fc005270 2023-07-04T18:50:57.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.651+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff61415c9b0 con 0x7ff5fc005270 2023-07-04T18:50:57.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.653+0000 7ff5fb7fe700 1 -- 172.21.15.77:0/2027861767 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff60c01da90 con 0x7ff5fc005270 2023-07-04T18:50:57.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 msgr2=0x7ff614149f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff60c004f90 tx=0x7ff60c007340 comp rx=0 tx=0).stop 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 shutdown_connections 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc005270 0x7ff614149f50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff614069e40 0x7ff61406a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 --2- 172.21.15.77:0/2027861767 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff614158020 0x7ff61414df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:57.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 >> 172.21.15.77:0/2027861767 conn(0x7ff61406af50 msgr2=0x7ff6140b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:57.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 shutdown_connections 2023-07-04T18:50:57.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:50:57.654+0000 7ff619039700 1 -- 172.21.15.77:0/2027861767 wait complete. 2023-07-04T18:50:57.657 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2023-07-04T18:50:57.667 INFO:tasks.ceph:check_enable_crimson: False 2023-07-04T18:50:57.667 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-07-04T18:50:57.668 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:57.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-07-04T18:50:57.689 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:57.687+0000 7f4aad45a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-04T18:50:57.693 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:57.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-07-04T18:50:57.754 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:57.752+0000 7f4aad45a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-04T18:50:57.759 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:57.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-07-04T18:50:57.825 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:50:57.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-07-04T18:50:57.829 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:57.827+0000 7f4aad45a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-04T18:50:57.891 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:57.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-07-04T18:50:57.909 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:57.908+0000 7f4aad45a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-04T18:50:57.919 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:57.919 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-07-04T18:50:57.988 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:57.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-07-04T18:50:58.045 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.043+0000 7f84ce4e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-04T18:50:58.057 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:50:58.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-07-04T18:50:58.065 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:58.062+0000 7f4aad45a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-04T18:50:58.092 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 1ece9a66-e00e-486d-860c-1ee57a7de146 0 2023-07-04T18:50:58.113 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.108+0000 7f84ce4e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-04T18:50:58.184 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.183+0000 7f84ce4e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-04T18:50:58.254 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.252+0000 7fcc70366700 1 Processor -- start 2023-07-04T18:50:58.255 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.252+0000 7fcc70366700 1 -- start start 2023-07-04T18:50:58.256 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc70366700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcc680b3330 0x7fcc680b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc70366700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680b3f60 con 0x7fcc6806cfc0 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc70366700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6806c4e0 con 0x7fcc681590b0 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc70366700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6806caa0 con 0x7fcc680b3330 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc6d901700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcc680b3330 0x7fcc680b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc6d901700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcc680b3330 0x7fcc680b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36222/0 (socket says 172.21.15.82:36222) 2023-07-04T18:50:58.257 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc6d901700 1 -- 172.21.15.82:0/552676515 learned_addr learned my addr 172.21.15.82:0/552676515 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.253+0000 7fcc6e102700 1 --1- 172.21.15.82:0/552676515 >> v1:172.21.15.82:6789/0 conn(0x7fcc681590b0 0x7fcc6815d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:35552/0 (socket says 172.21.15.82:35552) 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d901700 1 -- 172.21.15.82:0/552676515 >> v1:172.21.15.82:6789/0 conn(0x7fcc681590b0 legacy=0x7fcc6815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d901700 1 -- 172.21.15.82:0/552676515 >> v1:172.21.15.77:6789/0 conn(0x7fcc6806cfc0 legacy=0x7fcc6806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d901700 1 -- 172.21.15.82:0/552676515 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6806c7c0 con 0x7fcc680b3330 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d901700 1 --2- 172.21.15.82:0/552676515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc680b43f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc580098f0 tx=0x7fcc58009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edcb1bd52bf7d09f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.258 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d100700 1 -- 172.21.15.82:0/552676515 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5800b8a0 con 0x7fcc680b3330 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d100700 1 -- 172.21.15.82:0/552676515 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5800ba00 con 0x7fcc680b3330 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc6d100700 1 -- 172.21.15.82:0/552676515 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5800bd90 con 0x7fcc680b3330 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc70366700 1 -- 172.21.15.82:0/552676515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 msgr2=0x7fcc680b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.254+0000 7fcc70366700 1 --2- 172.21.15.82:0/552676515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc680b43f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc580098f0 tx=0x7fcc58009c00 comp rx=0 tx=0).stop 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.255+0000 7fcc70366700 1 -- 172.21.15.82:0/552676515 shutdown_connections 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.255+0000 7fcc70366700 1 --2- 172.21.15.82:0/552676515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc680b43f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.255+0000 7fcc70366700 1 -- 172.21.15.82:0/552676515 >> 172.21.15.82:0/552676515 conn(0x7fcc6806af50 msgr2=0x7fcc6806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.259 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.255+0000 7fcc70366700 1 -- 172.21.15.82:0/552676515 shutdown_connections 2023-07-04T18:50:58.260 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.255+0000 7fcc70366700 1 -- 172.21.15.82:0/552676515 wait complete. 2023-07-04T18:50:58.261 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.256+0000 7fcc70366700 1 Processor -- start 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.256+0000 7fcc70366700 1 -- start start 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.256+0000 7fcc70366700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 0x7fcc68166a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.256+0000 7fcc70366700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc681590b0 0x7fcc6815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680b1200 con 0x7fcc681590b0 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680b14e0 con 0x7fcc680b3330 2023-07-04T18:50:58.262 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680b17c0 con 0x7fcc6806cfc0 2023-07-04T18:50:58.263 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.263 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36228/0 (socket says 172.21.15.82:36228) 2023-07-04T18:50:58.263 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6e102700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 0x7fcc68166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 -- 172.21.15.82:0/3072185372 learned_addr learned my addr 172.21.15.82:0/3072185372 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6e903700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc681590b0 0x7fcc6815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 -- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 msgr2=0x7fcc68166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 0x7fcc68166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 -- 172.21.15.82:0/3072185372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc681590b0 msgr2=0x7fcc6815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc681590b0 0x7fcc6815a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.264 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 -- 172.21.15.82:0/3072185372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc58009550 con 0x7fcc680b3330 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6e102700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 0x7fcc68166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc6d901700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc58000c00 tx=0x7fcc58004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58004380 con 0x7fcc680b3330 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5801f070 con 0x7fcc680b3330 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc680b1aa0 con 0x7fcc680b3330 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc580036a0 con 0x7fcc680b3330 2023-07-04T18:50:58.265 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc680b22a0 con 0x7fcc680b3330 2023-07-04T18:50:58.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.257+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcc58021030 con 0x7fcc680b3330 2023-07-04T18:50:58.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.258+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcc5801a980 con 0x7fcc680b3330 2023-07-04T18:50:58.266 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.258+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6804e210 con 0x7fcc680b3330 2023-07-04T18:50:58.266 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.260+0000 7f84ce4e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-04T18:50:58.267 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.260+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fcc58015020 con 0x7fcc680b3330 2023-07-04T18:50:58.351 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.344+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1ece9a66-e00e-486d-860c-1ee57a7de146", "id": 0} v 0) v1 -- 0x7fcc6806b6f0 con 0x7fcc680b3330 2023-07-04T18:50:58.360 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.353+0000 7fcc5effd700 1 -- 172.21.15.82:0/3072185372 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ece9a66-e00e-486d-860c-1ee57a7de146", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcc58003ce0 con 0x7fcc680b3330 2023-07-04T18:50:58.361 INFO:teuthology.orchestra.run.smithi082.stdout:0 2023-07-04T18:50:58.362 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.354+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 msgr2=0x7fcc681413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.354+0000 7fcc70366700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc58000c00 tx=0x7fcc58004750 comp rx=0 tx=0).stop 2023-07-04T18:50:58.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 shutdown_connections 2023-07-04T18:50:58.363 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc681590b0 0x7fcc6815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcc6806cfc0 0x7fcc68166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.364 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 --2- 172.21.15.82:0/3072185372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc680b3330 0x7fcc681413a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 >> 172.21.15.82:0/3072185372 conn(0x7fcc6806af50 msgr2=0x7fcc6815b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 shutdown_connections 2023-07-04T18:50:58.365 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.355+0000 7fcc70366700 1 -- 172.21.15.82:0/3072185372 wait complete. 2023-07-04T18:50:58.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new c2fddd95-3531-4fc8-8749-f8e4d50b7b1f 1 2023-07-04T18:50:58.405 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.403+0000 7f84ce4e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-04T18:50:58.432 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:58.431+0000 7f4aad45a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-04T18:50:58.492 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.490+0000 7f6eaa637700 1 Processor -- start 2023-07-04T18:50:58.492 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.490+0000 7f6eaa637700 1 -- start start 2023-07-04T18:50:58.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6eaa637700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ea40adc40 0x7f6ea415d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6eaa637700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea406bd70 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6eaa637700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea41596b0 con 0x7f6ea41592e0 2023-07-04T18:50:58.494 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6eaa637700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea406b900 con 0x7f6ea40adc40 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ea40adc40 0x7f6ea415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ea40adc40 0x7f6ea415d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36234/0 (socket says 172.21.15.82:36234) 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 -- 172.21.15.82:0/2842091922 learned_addr learned my addr 172.21.15.82:0/2842091922 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea3fff700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6ea41592e0 0x7f6ea415cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:35564/0 (socket says 172.21.15.82:35564) 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 -- 172.21.15.82:0/2842091922 >> v1:172.21.15.82:6789/0 conn(0x7f6ea41592e0 legacy=0x7f6ea415cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 -- 172.21.15.82:0/2842091922 >> v1:172.21.15.77:6789/0 conn(0x7f6ea40b3ab0 legacy=0x7f6ea406fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.495 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 -- 172.21.15.82:0/2842091922 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea415da80 con 0x7f6ea40adc40 2023-07-04T18:50:58.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea8bd4700 1 --2- 172.21.15.82:0/2842091922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40adc40 0x7f6ea415d410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009e50 tx=0x7f6e9c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1973d834a31ffa15 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.491+0000 7f6ea2ffd700 1 -- 172.21.15.82:0/2842091922 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c00bbc0 con 0x7f6ea40adc40 2023-07-04T18:50:58.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6ea2ffd700 1 -- 172.21.15.82:0/2842091922 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c00bd20 con 0x7f6ea40adc40 2023-07-04T18:50:58.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6ea2ffd700 1 -- 172.21.15.82:0/2842091922 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0116a0 con 0x7f6ea40adc40 2023-07-04T18:50:58.496 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 -- 172.21.15.82:0/2842091922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40adc40 msgr2=0x7f6ea415d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2842091922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40adc40 0x7f6ea415d410 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009e50 tx=0x7f6e9c00b330 comp rx=0 tx=0).stop 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 -- 172.21.15.82:0/2842091922 shutdown_connections 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2842091922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40adc40 0x7f6ea415d410 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 -- 172.21.15.82:0/2842091922 >> 172.21.15.82:0/2842091922 conn(0x7f6ea4065bd0 msgr2=0x7f6ea4065fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 -- 172.21.15.82:0/2842091922 shutdown_connections 2023-07-04T18:50:58.497 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.492+0000 7f6eaa637700 1 -- 172.21.15.82:0/2842091922 wait complete. 2023-07-04T18:50:58.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.493+0000 7f6eaa637700 1 Processor -- start 2023-07-04T18:50:58.499 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 -- start start 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40adc40 0x7f6ea414c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 0x7f6ea41579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4158340 con 0x7f6ea40adc40 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4159f60 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6eaa637700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea415a240 con 0x7f6ea41592e0 2023-07-04T18:50:58.500 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea37fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea8bd4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 0x7f6ea41579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40adc40 0x7f6ea414c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea37fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36238/0 (socket says 172.21.15.82:36238) 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea37fe700 1 -- 172.21.15.82:0/2403147383 learned_addr learned my addr 172.21.15.82:0/2403147383 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.494+0000 7f6ea8bd4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 0x7f6ea41579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43296/0 (socket says 172.21.15.82:43296) 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 -- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 msgr2=0x7f6ea41579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.501 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 0x7f6ea41579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 -- 172.21.15.82:0/2403147383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40adc40 msgr2=0x7f6ea414c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40adc40 0x7f6ea414c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 -- 172.21.15.82:0/2403147383 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9c009b30 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea37fe700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c450 tx=0x7f6e9800c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980052d0 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98005430 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.502 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea415a520 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980038c0 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea414c780 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6e98003d20 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.495+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f6e98011d90 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.503 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.496+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea40af2c0 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.504 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.498+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f6e98017070 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.512 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:58.511+0000 7f4aad45a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-04T18:50:58.584 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.583+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c2fddd95-3531-4fc8-8749-f8e4d50b7b1f", "id": 1} v 0) v1 -- 0x7f6ea404e210 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.592 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.590+0000 7f6ea0ff9700 1 -- 172.21.15.82:0/2403147383 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2fddd95-3531-4fc8-8749-f8e4d50b7b1f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6e98011590 con 0x7f6ea40b3ab0 2023-07-04T18:50:58.593 INFO:teuthology.orchestra.run.smithi082.stdout:1 2023-07-04T18:50:58.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 msgr2=0x7f6ea4157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c450 tx=0x7f6e9800c760 comp rx=0 tx=0).stop 2023-07-04T18:50:58.594 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 shutdown_connections 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ea40adc40 0x7f6ea414c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ea41592e0 0x7f6ea41579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 --2- 172.21.15.82:0/2403147383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ea40b3ab0 0x7f6ea4157460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 >> 172.21.15.82:0/2403147383 conn(0x7f6ea4065bd0 msgr2=0x7f6ea415d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 shutdown_connections 2023-07-04T18:50:58.595 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.591+0000 7f6eaa637700 1 -- 172.21.15.82:0/2403147383 wait complete. 2023-07-04T18:50:58.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 9f71759a-fb46-4929-a09d-1a36386ac3bd 2 2023-07-04T18:50:58.648 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:58.646+0000 7f4aad45a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-04T18:50:58.725 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.723+0000 7f429a53d700 1 Processor -- start 2023-07-04T18:50:58.725 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.723+0000 7f429a53d700 1 -- start start 2023-07-04T18:50:58.727 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f429a53d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f429406ea90 0x7f429415d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f429a53d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f429a53d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429415da80 con 0x7f4294160a80 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f429a53d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940ac8a0 con 0x7f429406ea90 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f429a53d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429415d0b0 con 0x7f429406a550 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f4298ada700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.728 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f4298ada700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43300/0 (socket says 172.21.15.82:43300) 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f4298ada700 1 -- 172.21.15.82:0/1559473673 learned_addr learned my addr 172.21.15.82:0/1559473673 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.724+0000 7f4293fff700 1 --2- 172.21.15.82:0/1559473673 >> v2:172.21.15.77:3300/0 conn(0x7f429406ea90 0x7f429415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4298ada700 1 -- 172.21.15.82:0/1559473673 >> v2:172.21.15.77:3300/0 conn(0x7f429406ea90 msgr2=0x7f429415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4298ada700 1 --2- 172.21.15.82:0/1559473673 >> v2:172.21.15.77:3300/0 conn(0x7f429406ea90 0x7f429415d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4298ada700 1 -- 172.21.15.82:0/1559473673 >> v1:172.21.15.77:6789/0 conn(0x7f4294160a80 legacy=0x7f4294069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4298ada700 1 -- 172.21.15.82:0/1559473673 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429415ca80 con 0x7f429406a550 2023-07-04T18:50:58.729 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4298ada700 1 --2- 172.21.15.82:0/1559473673 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f428c009e50 tx=0x7f428c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0489ccedd3ea6f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4292ffd700 1 -- 172.21.15.82:0/1559473673 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c00e070 con 0x7f429406a550 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4292ffd700 1 -- 172.21.15.82:0/1559473673 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428c00b910 con 0x7f429406a550 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f4292ffd700 1 -- 172.21.15.82:0/1559473673 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c00bcc0 con 0x7f429406a550 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 -- 172.21.15.82:0/1559473673 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 msgr2=0x7f42940b2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 --2- 172.21.15.82:0/1559473673 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f428c009e50 tx=0x7f428c00b330 comp rx=0 tx=0).stop 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 -- 172.21.15.82:0/1559473673 shutdown_connections 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 --2- 172.21.15.82:0/1559473673 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42940b2db0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.730 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 --2- 172.21.15.82:0/1559473673 >> v2:172.21.15.77:3300/0 conn(0x7f429406ea90 0x7f429415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.731 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.725+0000 7f429a53d700 1 -- 172.21.15.82:0/1559473673 >> 172.21.15.82:0/1559473673 conn(0x7f429406af50 msgr2=0x7f429406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.731 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.726+0000 7f429a53d700 1 -- 172.21.15.82:0/1559473673 shutdown_connections 2023-07-04T18:50:58.731 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.726+0000 7f429a53d700 1 -- 172.21.15.82:0/1559473673 wait complete. 2023-07-04T18:50:58.733 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.727+0000 7f429a53d700 1 Processor -- start 2023-07-04T18:50:58.733 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.727+0000 7f429a53d700 1 -- start start 2023-07-04T18:50:58.733 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.733 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429406ea90 0x7f42940cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 0x7f4294157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cf510 con 0x7f429406ea90 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940d07a0 con 0x7f4294160a80 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cf700 con 0x7f429406a550 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43302/0 (socket says 172.21.15.82:43302) 2023-07-04T18:50:58.734 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4298ada700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 0x7f4294157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 -- 172.21.15.82:0/3978427425 learned_addr learned my addr 172.21.15.82:0/3978427425 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f42937fe700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429406ea90 0x7f42940cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 -- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 msgr2=0x7f4294157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 0x7f4294157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 -- 172.21.15.82:0/3978427425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429406ea90 msgr2=0x7f42940cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.735 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429406ea90 0x7f42940cefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 -- 172.21.15.82:0/3978427425 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c009b30 con 0x7f429406a550 2023-07-04T18:50:58.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4298ada700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 0x7f4294157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:58.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4293fff700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f428400e9b0 tx=0x7f428400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.736 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284009710 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428400ee50 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428400ca50 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42940cf9e0 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.728+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42940cffa0 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.729+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4284005a20 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.729+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f4284006700 con 0x7f429406a550 2023-07-04T18:50:58.737 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.729+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429404e210 con 0x7f429406a550 2023-07-04T18:50:58.738 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.731+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f4284021060 con 0x7f429406a550 2023-07-04T18:50:58.738 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:58.732+0000 7f4aad45a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-04T18:50:58.752 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.750+0000 7f84ce4e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-04T18:50:58.817 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.815+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9f71759a-fb46-4929-a09d-1a36386ac3bd", "id": 2} v 0) v1 -- 0x7f4294071570 con 0x7f429406a550 2023-07-04T18:50:58.825 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.823+0000 7f4290ff9700 1 -- 172.21.15.82:0/3978427425 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f71759a-fb46-4929-a09d-1a36386ac3bd", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4284017070 con 0x7f429406a550 2023-07-04T18:50:58.825 INFO:teuthology.orchestra.run.smithi082.stdout:2 2023-07-04T18:50:58.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 msgr2=0x7f42941f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.826 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f428400e9b0 tx=0x7f428400ecc0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 shutdown_connections 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4294160a80 0x7f4294157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f429406a550 0x7f42941f2ac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 --2- 172.21.15.82:0/3978427425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f429406ea90 0x7f42940cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 >> 172.21.15.82:0/3978427425 conn(0x7f429406af50 msgr2=0x7f429415fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 shutdown_connections 2023-07-04T18:50:58.827 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.824+0000 7f429a53d700 1 -- 172.21.15.82:0/3978427425 wait complete. 2023-07-04T18:50:58.828 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.826+0000 7f84ce4e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-04T18:50:58.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new fe8befe0-56b7-4f0c-bc08-910c2ec4637a 3 2023-07-04T18:50:58.943 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:58.941+0000 7f84ce4e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-04T18:50:58.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.957+0000 7fc96687c700 1 Processor -- start 2023-07-04T18:50:58.959 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc96687c700 1 -- start start 2023-07-04T18:50:58.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc96687c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc96687c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96015f180 con 0x7fc9601440b0 2023-07-04T18:50:58.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc96687c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600cee30 con 0x7fc9600ce8c0 2023-07-04T18:50:58.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc96687c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600cf110 con 0x7fc9600d01d0 2023-07-04T18:50:58.961 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc95f7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc95f7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43310/0 (socket says 172.21.15.82:43310) 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.958+0000 7fc95f7fe700 1 -- 172.21.15.82:0/1176418543 learned_addr learned my addr 172.21.15.82:0/1176418543 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95effd700 1 -- 172.21.15.82:0/1176418543 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254546666 0 0) 0x7fc9600cee30 con 0x7fc9600ce8c0 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95f7fe700 1 -- 172.21.15.82:0/1176418543 >> v1:172.21.15.82:6789/0 conn(0x7fc9600ce8c0 legacy=0x7fc9600cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95f7fe700 1 -- 172.21.15.82:0/1176418543 >> v1:172.21.15.77:6789/0 conn(0x7fc9601440b0 legacy=0x7fc9600cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95f7fe700 1 -- 172.21.15.82:0/1176418543 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600cf3f0 con 0x7fc9600d01d0 2023-07-04T18:50:58.962 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95f7fe700 1 --2- 172.21.15.82:0/1176418543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc954009920 tx=0x7fc954009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c7245a512b8f33 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.963 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95effd700 1 -- 172.21.15.82:0/1176418543 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004230 con 0x7fc9600d01d0 2023-07-04T18:50:58.963 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95effd700 1 -- 172.21.15.82:0/1176418543 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc954004390 con 0x7fc9600d01d0 2023-07-04T18:50:58.963 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.959+0000 7fc95effd700 1 -- 172.21.15.82:0/1176418543 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954009e70 con 0x7fc9600d01d0 2023-07-04T18:50:58.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 -- 172.21.15.82:0/1176418543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 msgr2=0x7fc96015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 --2- 172.21.15.82:0/1176418543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc954009920 tx=0x7fc954009c30 comp rx=0 tx=0).stop 2023-07-04T18:50:58.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 -- 172.21.15.82:0/1176418543 shutdown_connections 2023-07-04T18:50:58.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 --2- 172.21.15.82:0/1176418543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9600d01d0 0x7fc96015ec40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.966 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 -- 172.21.15.82:0/1176418543 >> 172.21.15.82:0/1176418543 conn(0x7fc96006d900 msgr2=0x7fc96006dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 -- 172.21.15.82:0/1176418543 shutdown_connections 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.960+0000 7fc96687c700 1 -- 172.21.15.82:0/1176418543 wait complete. 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.961+0000 7fc96687c700 1 Processor -- start 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 -- start start 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600d01d0 0x7fc96015d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:58.967 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96015aa10 con 0x7fc9600d01d0 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f0db0 con 0x7fc9600ce8c0 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.962+0000 7fc96687c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601f1090 con 0x7fc9601440b0 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc964e19700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36258/0 (socket says 172.21.15.82:36258) 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc964e19700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43318/0 (socket says 172.21.15.82:43318) 2023-07-04T18:50:58.968 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 -- 172.21.15.82:0/900394060 learned_addr learned my addr 172.21.15.82:0/900394060 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95f7fe700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600d01d0 0x7fc96015d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 -- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 msgr2=0x7fc9601f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 -- 172.21.15.82:0/900394060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600d01d0 msgr2=0x7fc96015d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600d01d0 0x7fc96015d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 -- 172.21.15.82:0/900394060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954009550 con 0x7fc9600ce8c0 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc964e19700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:58.969 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95ffff700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c700 tx=0x7fc95000ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9500054e0 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc950003680 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96015ac60 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950003a30 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9601f19c0 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc95001c440 con 0x7fc9600ce8c0 2023-07-04T18:50:58.970 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.963+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc950012890 con 0x7fc9600ce8c0 2023-07-04T18:50:58.971 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.964+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96015b180 con 0x7fc9600ce8c0 2023-07-04T18:50:58.971 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:58.966+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fc96015b180 con 0x7fc9600ce8c0 2023-07-04T18:50:59.024 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:59.023+0000 7f84ce4e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-04T18:50:59.051 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.050+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fe8befe0-56b7-4f0c-bc08-910c2ec4637a", "id": 3} v 0) v1 -- 0x7fc96004e210 con 0x7fc9600ce8c0 2023-07-04T18:50:59.059 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.057+0000 7fc95cff9700 1 -- 172.21.15.82:0/900394060 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe8befe0-56b7-4f0c-bc08-910c2ec4637a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc950017070 con 0x7fc9600ce8c0 2023-07-04T18:50:59.060 INFO:teuthology.orchestra.run.smithi082.stdout:3 2023-07-04T18:50:59.061 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 msgr2=0x7fc9601f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.061 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c700 tx=0x7fc95000ca10 comp rx=0 tx=0).stop 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 shutdown_connections 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc9601440b0 0x7fc9601f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc9600ce8c0 0x7fc9601f1480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 --2- 172.21.15.82:0/900394060 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600d01d0 0x7fc96015d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.058+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 >> 172.21.15.82:0/900394060 conn(0x7fc96006d900 msgr2=0x7fc9600c6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.059+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 shutdown_connections 2023-07-04T18:50:59.062 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.059+0000 7fc96687c700 1 -- 172.21.15.82:0/900394060 wait complete. 2023-07-04T18:50:59.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 8dada326-1025-469e-93b0-0e7e4aa3cdf0 4 2023-07-04T18:50:59.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.190+0000 7f3f77332700 1 Processor -- start 2023-07-04T18:50:59.192 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.191+0000 7f3f77332700 1 -- start start 2023-07-04T18:50:59.197 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f77332700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f77332700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f77332700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700cd300 con 0x7f3f7015c510 2023-07-04T18:50:59.198 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f77332700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f70070940 con 0x7f3f700704c0 2023-07-04T18:50:59.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f77332700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700ca710 con 0x7f3f700ca130 2023-07-04T18:50:59.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f758cf700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.199 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f758cf700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36268/0 (socket says 172.21.15.82:36268) 2023-07-04T18:50:59.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43322/0 (socket says 172.21.15.82:43322) 2023-07-04T18:50:59.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f758cf700 1 -- 172.21.15.82:0/1082681350 learned_addr learned my addr 172.21.15.82:0/1082681350 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.200 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 -- 172.21.15.82:0/1082681350 >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 msgr2=0x7f3f700cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:59.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 --2- 172.21.15.82:0/1082681350 >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 -- 172.21.15.82:0/1082681350 >> v1:172.21.15.77:6789/0 conn(0x7f3f7015c510 legacy=0x7f3f7006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:59.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f748cd700 1 -- 172.21.15.82:0/1082681350 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700ca9f0 con 0x7f3f700ca130 2023-07-04T18:50:59.201 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.192+0000 7f3f758cf700 1 --2- 172.21.15.82:0/1082681350 >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f748cd700 1 --2- 172.21.15.82:0/1082681350 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009840 tx=0x7f3f60009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8df50cb36e84826 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f67fff700 1 -- 172.21.15.82:0/1082681350 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60004230 con 0x7f3f700ca130 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f67fff700 1 -- 172.21.15.82:0/1082681350 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60004390 con 0x7f3f700ca130 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f67fff700 1 -- 172.21.15.82:0/1082681350 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f600036a0 con 0x7f3f700ca130 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 -- 172.21.15.82:0/1082681350 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 msgr2=0x7f3f7006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.202 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 --2- 172.21.15.82:0/1082681350 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009840 tx=0x7f3f60009b50 comp rx=0 tx=0).stop 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 -- 172.21.15.82:0/1082681350 shutdown_connections 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 --2- 172.21.15.82:0/1082681350 >> v2:172.21.15.82:3300/0 conn(0x7f3f700704c0 0x7f3f700cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 --2- 172.21.15.82:0/1082681350 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f700ca130 0x7f3f7006ff80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.193+0000 7f3f77332700 1 -- 172.21.15.82:0/1082681350 >> 172.21.15.82:0/1082681350 conn(0x7f3f700cf0d0 msgr2=0x7f3f700cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.194+0000 7f3f77332700 1 -- 172.21.15.82:0/1082681350 shutdown_connections 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.194+0000 7f3f77332700 1 -- 172.21.15.82:0/1082681350 wait complete. 2023-07-04T18:50:59.203 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 Processor -- start 2023-07-04T18:50:59.204 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 -- start start 2023-07-04T18:50:59.204 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f4aad45a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-04T18:50:59.206 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 0x7f3f7006d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f700ca130 0x7f3f7009bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7006d7c0 con 0x7f3f700ca130 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c3a60 con 0x7f3f700704c0 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.195+0000 7f3f77332700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c3d40 con 0x7f3f7015c510 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f750ce700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 0x7f3f7006d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.207 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43330/0 (socket says 172.21.15.82:43330) 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 -- 172.21.15.82:0/971087566 learned_addr learned my addr 172.21.15.82:0/971087566 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f748cd700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f700ca130 0x7f3f7009bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 -- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 msgr2=0x7f3f7006d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 0x7f3f7006d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 -- 172.21.15.82:0/971087566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f700ca130 msgr2=0x7f3f7009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f700ca130 0x7f3f7009bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.208 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 -- 172.21.15.82:0/971087566 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68009b30 con 0x7f3f7015c510 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f750ce700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 0x7f3f7006d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f758cf700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800efc0 tx=0x7f3f6800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68018070 con 0x7f3f7015c510 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6800f040 con 0x7f3f7015c510 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f600094a0 con 0x7f3f7015c510 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68003c20 con 0x7f3f7015c510 2023-07-04T18:50:59.209 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f700c4300 con 0x7f3f7015c510 2023-07-04T18:50:59.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.196+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3f6800cb80 con 0x7f3f7015c510 2023-07-04T18:50:59.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.197+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f3f6800ce30 con 0x7f3f7015c510 2023-07-04T18:50:59.210 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.197+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7004e210 con 0x7f3f7015c510 2023-07-04T18:50:59.212 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.199+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f3f680047a0 con 0x7f3f7015c510 2023-07-04T18:50:59.282 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.281+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8dada326-1025-469e-93b0-0e7e4aa3cdf0", "id": 4} v 0) v1 -- 0x7f3f700b5050 con 0x7f3f7015c510 2023-07-04T18:50:59.291 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.289+0000 7f3f65ffb700 1 -- 172.21.15.82:0/971087566 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8dada326-1025-469e-93b0-0e7e4aa3cdf0", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3f6802c0b0 con 0x7f3f7015c510 2023-07-04T18:50:59.291 INFO:teuthology.orchestra.run.smithi082.stdout:4 2023-07-04T18:50:59.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 msgr2=0x7f3f7009c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800efc0 tx=0x7f3f6800c3a0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 shutdown_connections 2023-07-04T18:50:59.292 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3f7015c510 0x7f3f7009c200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f700704c0 0x7f3f7006d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 --2- 172.21.15.82:0/971087566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f700ca130 0x7f3f7009bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 >> 172.21.15.82:0/971087566 conn(0x7f3f700cf0d0 msgr2=0x7f3f7015b800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 shutdown_connections 2023-07-04T18:50:59.293 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.290+0000 7f3f77332700 1 -- 172.21.15.82:0/971087566 wait complete. 2023-07-04T18:50:59.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 53f40546-3ced-4dcd-8ca1-b3ca6a0a576d 5 2023-07-04T18:50:59.422 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.420+0000 7fef3de39700 1 Processor -- start 2023-07-04T18:50:59.423 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.420+0000 7fef3de39700 1 -- start start 2023-07-04T18:50:59.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef3de39700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 0x7fef3815d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef3de39700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fef38160a80 0x7fef38069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.425 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef3de39700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3815da80 con 0x7fef3806a380 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef3de39700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3806a800 con 0x7fef38160a80 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef3de39700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef380ac8a0 con 0x7fef3806ea90 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef37fff700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fef3806a380 0x7fef380b2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:35612/0 (socket says 172.21.15.82:35612) 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef377fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 0x7fef3815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef37fff700 1 -- 172.21.15.82:0/3779848960 learned_addr learned my addr 172.21.15.82:0/3779848960 (peer_addr_for_me v1:172.21.15.82:0/0) 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.421+0000 7fef36ffd700 1 --2- 172.21.15.82:0/3779848960 >> v2:172.21.15.77:3300/0 conn(0x7fef38160a80 0x7fef38069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.426 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef367fc700 1 -- 172.21.15.82:0/3779848960 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862635028 0 0) 0x7fef30001490 con 0x7fef3806a380 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef377fe700 1 -- 172.21.15.82:0/3779848960 >> v2:172.21.15.77:3300/0 conn(0x7fef38160a80 msgr2=0x7fef38069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef377fe700 1 --2- 172.21.15.82:0/3779848960 >> v2:172.21.15.77:3300/0 conn(0x7fef38160a80 0x7fef38069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef377fe700 1 -- 172.21.15.82:0/3779848960 >> v1:172.21.15.82:6789/0 conn(0x7fef3806a380 legacy=0x7fef380b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef377fe700 1 -- 172.21.15.82:0/3779848960 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3815d0b0 con 0x7fef3806ea90 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef377fe700 1 --2- 172.21.15.82:0/3779848960 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 0x7fef3815d540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef28009840 tx=0x7fef28009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87854786605e556 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef367fc700 1 -- 172.21.15.82:0/3779848960 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28011830 con 0x7fef3806ea90 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef367fc700 1 -- 172.21.15.82:0/3779848960 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef28011990 con 0x7fef3806ea90 2023-07-04T18:50:59.427 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef367fc700 1 -- 172.21.15.82:0/3779848960 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28011d20 con 0x7fef3806ea90 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef3de39700 1 -- 172.21.15.82:0/3779848960 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 msgr2=0x7fef3815d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.422+0000 7fef3de39700 1 --2- 172.21.15.82:0/3779848960 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 0x7fef3815d540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef28009840 tx=0x7fef28009b50 comp rx=0 tx=0).stop 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 -- 172.21.15.82:0/3779848960 shutdown_connections 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 --2- 172.21.15.82:0/3779848960 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806ea90 0x7fef3815d540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 --2- 172.21.15.82:0/3779848960 >> v2:172.21.15.77:3300/0 conn(0x7fef38160a80 0x7fef38069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 -- 172.21.15.82:0/3779848960 >> 172.21.15.82:0/3779848960 conn(0x7fef3806af50 msgr2=0x7fef3806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.428 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 -- 172.21.15.82:0/3779848960 shutdown_connections 2023-07-04T18:50:59.429 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.423+0000 7fef3de39700 1 -- 172.21.15.82:0/3779848960 wait complete. 2023-07-04T18:50:59.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.424+0000 7fef3de39700 1 Processor -- start 2023-07-04T18:50:59.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.424+0000 7fef3de39700 1 -- start start 2023-07-04T18:50:59.431 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806a380 0x7fef381e6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef38160a80 0x7fef380b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381e8540 con 0x7fef38160a80 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381e78c0 con 0x7fef3806ea90 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef381e7ba0 con 0x7fef3806a380 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36284/0 (socket says 172.21.15.82:36284) 2023-07-04T18:50:59.432 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 -- 172.21.15.82:0/2104800280 learned_addr learned my addr 172.21.15.82:0/2104800280 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef377fe700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806a380 0x7fef381e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef37fff700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef38160a80 0x7fef380b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 -- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806a380 msgr2=0x7fef381e6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806a380 0x7fef381e6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.433 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 -- 172.21.15.82:0/2104800280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef38160a80 msgr2=0x7fef380b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef38160a80 0x7fef380b0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 -- 172.21.15.82:0/2104800280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280094a0 con 0x7fef3806ea90 2023-07-04T18:50:59.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef36ffd700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00e9b0 tx=0x7fef2c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c009710 con 0x7fef3806ea90 2023-07-04T18:50:59.434 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c00ee50 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef381e7e80 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.425+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00ca50 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.426+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef381e6b50 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.426+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fef2c005a20 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.426+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fef2c006700 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.426+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef380b2490 con 0x7fef3806ea90 2023-07-04T18:50:59.435 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.428+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fef2c021060 con 0x7fef3806ea90 2023-07-04T18:50:59.461 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:50:59.460+0000 7f84ce4e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-04T18:50:59.515 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.513+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd new", "uuid": "53f40546-3ced-4dcd-8ca1-b3ca6a0a576d", "id": 5} v 0) v1 -- 0x7fef3804e210 con 0x7fef3806ea90 2023-07-04T18:50:59.522 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.521+0000 7fef1ffff700 1 -- 172.21.15.82:0/2104800280 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53f40546-3ced-4dcd-8ca1-b3ca6a0a576d", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fef2c017070 con 0x7fef3806ea90 2023-07-04T18:50:59.523 INFO:teuthology.orchestra.run.smithi082.stdout:5 2023-07-04T18:50:59.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 msgr2=0x7fef380cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.524 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00e9b0 tx=0x7fef2c00ecc0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 shutdown_connections 2023-07-04T18:50:59.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef38160a80 0x7fef380b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fef3806a380 0x7fef381e6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 --2- 172.21.15.82:0/2104800280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fef3806ea90 0x7fef380cd9e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.525 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 >> 172.21.15.82:0/2104800280 conn(0x7fef3806af50 msgr2=0x7fef3806c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.526 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 shutdown_connections 2023-07-04T18:50:59.526 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.522+0000 7fef3de39700 1 -- 172.21.15.82:0/2104800280 wait complete. 2023-07-04T18:50:59.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6 6 2023-07-04T18:50:59.684 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.682+0000 7f2fd1ff0700 1 Processor -- start 2023-07-04T18:50:59.684 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.682+0000 7f2fd1ff0700 1 -- start start 2023-07-04T18:50:59.694 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fd1ff0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 0x7f2fcc15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.694 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fd1ff0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2fcc0b3330 0x7f2fcc0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.695 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fd1ff0700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc0b3f60 con 0x7f2fcc06cfc0 2023-07-04T18:50:59.695 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fd1ff0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc06caa0 con 0x7f2fcc0b3330 2023-07-04T18:50:59.695 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fd1ff0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15dc10 con 0x7f2fcc1590b0 2023-07-04T18:50:59.696 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 0x7f2fcc15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.697 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcbfff700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f2fcc06cfc0 0x7f2fcc159480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:35622/0 (socket says 172.21.15.82:35622) 2023-07-04T18:50:59.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcbfff700 1 -- 172.21.15.82:0/683879084 learned_addr learned my addr 172.21.15.82:0/683879084 (peer_addr_for_me v1:172.21.15.82:0/0) 2023-07-04T18:50:59.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcaffd700 1 --2- 172.21.15.82:0/683879084 >> v2:172.21.15.77:3300/0 conn(0x7f2fcc0b3330 0x7f2fcc0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.698 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 -- 172.21.15.82:0/683879084 >> v2:172.21.15.77:3300/0 conn(0x7f2fcc0b3330 msgr2=0x7f2fcc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:59.699 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 --2- 172.21.15.82:0/683879084 >> v2:172.21.15.77:3300/0 conn(0x7f2fcc0b3330 0x7f2fcc0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.699 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 -- 172.21.15.82:0/683879084 >> v1:172.21.15.82:6789/0 conn(0x7f2fcc06cfc0 legacy=0x7f2fcc159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.699 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 -- 172.21.15.82:0/683879084 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc06c4e0 con 0x7f2fcc1590b0 2023-07-04T18:50:59.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fca7fc700 1 -- 172.21.15.82:0/683879084 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500649304 0 0) 0x7f2fcc0b3f60 con 0x7f2fcc06cfc0 2023-07-04T18:50:59.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fcb7fe700 1 --2- 172.21.15.82:0/683879084 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 0x7f2fcc15d6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009840 tx=0x7f2fbc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6dba5ab2173fd4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fca7fc700 1 -- 172.21.15.82:0/683879084 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc011830 con 0x7f2fcc1590b0 2023-07-04T18:50:59.700 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fca7fc700 1 -- 172.21.15.82:0/683879084 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc011990 con 0x7f2fcc1590b0 2023-07-04T18:50:59.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.683+0000 7f2fca7fc700 1 -- 172.21.15.82:0/683879084 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc011d20 con 0x7f2fcc1590b0 2023-07-04T18:50:59.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/683879084 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 msgr2=0x7f2fcc15d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/683879084 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 0x7f2fcc15d6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009840 tx=0x7f2fbc009b50 comp rx=0 tx=0).stop 2023-07-04T18:50:59.701 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/683879084 shutdown_connections 2023-07-04T18:50:59.702 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/683879084 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc1590b0 0x7f2fcc15d6d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.702 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/683879084 >> v2:172.21.15.77:3300/0 conn(0x7f2fcc0b3330 0x7f2fcc0b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.702 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/683879084 >> 172.21.15.82:0/683879084 conn(0x7f2fcc06af50 msgr2=0x7f2fcc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.703 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/683879084 shutdown_connections 2023-07-04T18:50:59.703 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.684+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/683879084 wait complete. 2023-07-04T18:50:59.703 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.685+0000 7f2fd1ff0700 1 Processor -- start 2023-07-04T18:50:59.703 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.685+0000 7f2fd1ff0700 1 -- start start 2023-07-04T18:50:59.704 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 0x7f2fcc14a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.704 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.705 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 0x7f2fcc15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.705 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1f2ef0 con 0x7f2fcc06cfc0 2023-07-04T18:50:59.705 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc14ac20 con 0x7f2fcc1590b0 2023-07-04T18:50:59.705 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fd1ff0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1f2c10 con 0x7f2fcc0b3330 2023-07-04T18:50:59.706 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fcaffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.707 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fcbfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 0x7f2fcc15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.707 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fcaffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.82:43352/0 (socket says 172.21.15.82:43352) 2023-07-04T18:50:59.708 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fcaffd700 1 -- 172.21.15.82:0/1831070988 learned_addr learned my addr 172.21.15.82:0/1831070988 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.708 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.686+0000 7f2fcb7fe700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 0x7f2fcc14a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.708 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 -- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 msgr2=0x7f2fcc15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.709 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 0x7f2fcc15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.709 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 -- 172.21.15.82:0/1831070988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 msgr2=0x7f2fcc14a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.709 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 0x7f2fcc14a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.710 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 -- 172.21.15.82:0/1831070988 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0094a0 con 0x7f2fcc0b3330 2023-07-04T18:50:59.710 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcbfff700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 0x7f2fcc15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:59.710 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcb7fe700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 0x7f2fcc14a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:59.711 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fcaffd700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000e9b0 tx=0x7f2fc000ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.711 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0009710 con 0x7f2fcc0b3330 2023-07-04T18:50:59.711 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc000ee50 con 0x7f2fcc0b3330 2023-07-04T18:50:59.712 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc14aff0 con 0x7f2fcc0b3330 2023-07-04T18:50:59.712 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc000ca50 con 0x7f2fcc0b3330 2023-07-04T18:50:59.712 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc14b5b0 con 0x7f2fcc0b3330 2023-07-04T18:50:59.712 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fc0013470 con 0x7f2fcc0b3330 2023-07-04T18:50:59.712 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.687+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f2fc001cb80 con 0x7f2fcc0b3330 2023-07-04T18:50:59.713 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.688+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc073790 con 0x7f2fcc0b3330 2023-07-04T18:50:59.713 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.690+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f2fc001ce60 con 0x7f2fcc0b3330 2023-07-04T18:50:59.774 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.772+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6", "id": 6} v 0) v1 -- 0x7f2fcc14b920 con 0x7f2fcc0b3330 2023-07-04T18:50:59.782 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.781+0000 7f2fb3fff700 1 -- 172.21.15.82:0/1831070988 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2fc0021060 con 0x7f2fcc0b3330 2023-07-04T18:50:59.786 INFO:teuthology.orchestra.run.smithi082.stdout:6 2023-07-04T18:50:59.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 msgr2=0x7f2fcc1efaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000e9b0 tx=0x7f2fc000ecc0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.788 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 shutdown_connections 2023-07-04T18:50:59.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2fcc06cfc0 0x7f2fcc14a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2fcc1590b0 0x7f2fcc15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 --2- 172.21.15.82:0/1831070988 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2fcc0b3330 0x7f2fcc1efaf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.789 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 >> 172.21.15.82:0/1831070988 conn(0x7f2fcc06af50 msgr2=0x7f2fcc0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 shutdown_connections 2023-07-04T18:50:59.790 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.782+0000 7f2fd1ff0700 1 -- 172.21.15.82:0/1831070988 wait complete. 2023-07-04T18:50:59.791 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:59.786+0000 7f4aad45a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-04T18:50:59.795 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 453bc1b4-29d7-485e-9771-38679a8bc1ad 7 2023-07-04T18:50:59.918 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.916+0000 7fab4ddea700 1 Processor -- start 2023-07-04T18:50:59.918 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.916+0000 7fab4ddea700 1 -- start start 2023-07-04T18:50:59.920 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab4ddea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab4806ec20 0x7fab4815d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab4ddea700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fab48160a60 0x7fab48069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab4ddea700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815da60 con 0x7fab4806a380 2023-07-04T18:50:59.921 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab4ddea700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4806eff0 con 0x7fab4806ec20 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab4ddea700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480ac8a0 con 0x7fab48160a60 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab46ffd700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fab48160a60 0x7fab48069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab46ffd700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fab48160a60 0x7fab48069e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36302/0 (socket says 172.21.15.82:36302) 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.917+0000 7fab46ffd700 1 -- 172.21.15.82:0/3056686774 learned_addr learned my addr 172.21.15.82:0/3056686774 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab477fe700 1 --2- 172.21.15.82:0/3056686774 >> v2:172.21.15.77:3300/0 conn(0x7fab4806ec20 0x7fab4815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab46ffd700 1 -- 172.21.15.82:0/3056686774 >> v2:172.21.15.77:3300/0 conn(0x7fab4806ec20 msgr2=0x7fab4815d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:50:59.922 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab46ffd700 1 --2- 172.21.15.82:0/3056686774 >> v2:172.21.15.77:3300/0 conn(0x7fab4806ec20 0x7fab4815d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab46ffd700 1 -- 172.21.15.82:0/3056686774 >> v1:172.21.15.77:6789/0 conn(0x7fab4806a380 legacy=0x7fab480b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab46ffd700 1 -- 172.21.15.82:0/3056686774 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4815d090 con 0x7fab48160a60 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab46ffd700 1 --2- 172.21.15.82:0/3056686774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48160a60 0x7fab48069e40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab480cc040 tx=0x7fab3c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b60e32bde7bb3 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab467fc700 1 -- 172.21.15.82:0/3056686774 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c011830 con 0x7fab48160a60 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab467fc700 1 -- 172.21.15.82:0/3056686774 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c011990 con 0x7fab48160a60 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.918+0000 7fab467fc700 1 -- 172.21.15.82:0/3056686774 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c011d20 con 0x7fab48160a60 2023-07-04T18:50:59.923 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 -- 172.21.15.82:0/3056686774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48160a60 msgr2=0x7fab48069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 --2- 172.21.15.82:0/3056686774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48160a60 0x7fab48069e40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fab480cc040 tx=0x7fab3c009bd0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 -- 172.21.15.82:0/3056686774 shutdown_connections 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 --2- 172.21.15.82:0/3056686774 >> v2:172.21.15.77:3300/0 conn(0x7fab4806ec20 0x7fab4815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 --2- 172.21.15.82:0/3056686774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48160a60 0x7fab48069e40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 -- 172.21.15.82:0/3056686774 >> 172.21.15.82:0/3056686774 conn(0x7fab4806af50 msgr2=0x7fab4806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 -- 172.21.15.82:0/3056686774 shutdown_connections 2023-07-04T18:50:59.924 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.919+0000 7fab4ddea700 1 -- 172.21.15.82:0/3056686774 wait complete. 2023-07-04T18:50:59.926 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.920+0000 7fab4ddea700 1 Processor -- start 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.920+0000 7fab4ddea700 1 -- start start 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4806a380 0x7fab481505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 0x7fab48151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4815b620 con 0x7fab4806a380 2023-07-04T18:50:59.927 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48150af0 con 0x7fab4806ec20 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab4ddea700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab48157ce0 con 0x7fab48160a60 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab46ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab46ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:36308/0 (socket says 172.21.15.82:36308) 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab46ffd700 1 -- 172.21.15.82:0/823531811 learned_addr learned my addr 172.21.15.82:0/823531811 (peer_addr_for_me v2:172.21.15.82:0/0) 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 0x7fab48151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab477fe700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4806a380 0x7fab481505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:50:59.928 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 -- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 msgr2=0x7fab4815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 -- 172.21.15.82:0/823531811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4806a380 msgr2=0x7fab481505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4806a380 0x7fab481505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 -- 172.21.15.82:0/823531811 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c009550 con 0x7fab48160a60 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab46ffd700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.921+0000 7fab47fff700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 0x7fab48151d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab4000cae0 tx=0x7fab4000cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40003c00 con 0x7fab48160a60 2023-07-04T18:50:59.929 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab40003d60 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000aa20 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4815ad80 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4815b340 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fab400050d0 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.922+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fab400124c0 con 0x7fab48160a60 2023-07-04T18:50:59.930 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.923+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48073790 con 0x7fab48160a60 2023-07-04T18:50:59.931 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:50:59.925+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fab4815da60 con 0x7fab48160a60 2023-07-04T18:50:59.958 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:50:59.957+0000 7f4aad45a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-04T18:51:00.020 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:00.019+0000 7f84ce4e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-04T18:51:00.021 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.019+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd new", "uuid": "453bc1b4-29d7-485e-9771-38679a8bc1ad", "id": 7} v 0) v1 -- 0x7fab4804e210 con 0x7fab48160a60 2023-07-04T18:51:00.029 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.027+0000 7fab2ffff700 1 -- 172.21.15.82:0/823531811 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "453bc1b4-29d7-485e-9771-38679a8bc1ad", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fab40005380 con 0x7fab48160a60 2023-07-04T18:51:00.030 INFO:teuthology.orchestra.run.smithi082.stdout:7 2023-07-04T18:51:00.032 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 msgr2=0x7fab48151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.032 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 0x7fab48151d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab4000cae0 tx=0x7fab4000cdf0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 shutdown_connections 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4806a380 0x7fab481505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab48160a60 0x7fab48151d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 --2- 172.21.15.82:0/823531811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4806ec20 0x7fab4815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 >> 172.21.15.82:0/823531811 conn(0x7fab4806af50 msgr2=0x7fab4806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 shutdown_connections 2023-07-04T18:51:00.033 INFO:teuthology.orchestra.run.smithi082.stderr:2023-07-04T18:51:00.028+0000 7fab4ddea700 1 -- 172.21.15.82:0/823531811 wait complete. 2023-07-04T18:51:00.042 INFO:tasks.ceph.osd.0:Restarting daemon 2023-07-04T18:51:00.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-07-04T18:51:00.045 INFO:tasks.ceph.osd.0:Started 2023-07-04T18:51:00.046 INFO:tasks.ceph.osd.1:Restarting daemon 2023-07-04T18:51:00.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-07-04T18:51:00.049 INFO:tasks.ceph.osd.1:Started 2023-07-04T18:51:00.049 INFO:tasks.ceph.osd.2:Restarting daemon 2023-07-04T18:51:00.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-07-04T18:51:00.053 INFO:tasks.ceph.osd.2:Started 2023-07-04T18:51:00.053 INFO:tasks.ceph.osd.3:Restarting daemon 2023-07-04T18:51:00.053 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-07-04T18:51:00.058 INFO:tasks.ceph.osd.3:Started 2023-07-04T18:51:00.059 INFO:tasks.ceph.osd.4:Restarting daemon 2023-07-04T18:51:00.059 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-07-04T18:51:00.062 INFO:tasks.ceph.osd.4:Started 2023-07-04T18:51:00.062 INFO:tasks.ceph.osd.5:Restarting daemon 2023-07-04T18:51:00.063 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-07-04T18:51:00.067 INFO:tasks.ceph.osd.5:Started 2023-07-04T18:51:00.068 INFO:tasks.ceph.osd.6:Restarting daemon 2023-07-04T18:51:00.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-07-04T18:51:00.075 INFO:tasks.ceph.osd.6:Started 2023-07-04T18:51:00.075 INFO:tasks.ceph.osd.7:Restarting daemon 2023-07-04T18:51:00.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-07-04T18:51:00.078 INFO:tasks.ceph.osd.7:Started 2023-07-04T18:51:00.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:00.213 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:00.212+0000 7f84ce4e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-04T18:51:00.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.223+0000 7fd91ce5c700 1 Processor -- start 2023-07-04T18:51:00.225 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.223+0000 7fd91ce5c700 1 -- start start 2023-07-04T18:51:00.226 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91ce5c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd91815ca10 0x7fd918070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91ce5c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fd918070db0 0x7fd9180cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91ce5c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9180cf830 con 0x7fd91815c640 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91ce5c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180711d0 con 0x7fd91815ca10 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91ce5c700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180717f0 con 0x7fd918070db0 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd91815ca10 0x7fd918070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:00.228 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd91815ca10 0x7fd918070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43322/0 (socket says 172.21.15.77:43322) 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 -- 172.21.15.77:0/278758766 learned_addr learned my addr 172.21.15.77:0/278758766 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd916d9d700 1 --2- 172.21.15.77:0/278758766 >> v2:172.21.15.82:3300/0 conn(0x7fd918070db0 0x7fd9180cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 -- 172.21.15.77:0/278758766 >> v2:172.21.15.82:3300/0 conn(0x7fd918070db0 msgr2=0x7fd9180cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 --2- 172.21.15.77:0/278758766 >> v2:172.21.15.82:3300/0 conn(0x7fd918070db0 0x7fd9180cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 -- 172.21.15.77:0/278758766 >> v1:172.21.15.82:6789/0 conn(0x7fd91815c640 legacy=0x7fd9180cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 -- 172.21.15.77:0/278758766 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180cd290 con 0x7fd91815ca10 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd915d9b700 1 --2- 172.21.15.77:0/278758766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815ca10 0x7fd918070870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd9180705f0 tx=0x7fd90c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aad4d27c495ddb5 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:00.229 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91559a700 1 -- 172.21.15.77:0/278758766 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c011830 con 0x7fd91815ca10 2023-07-04T18:51:00.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91559a700 1 -- 172.21.15.77:0/278758766 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c011990 con 0x7fd91815ca10 2023-07-04T18:51:00.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.224+0000 7fd91559a700 1 -- 172.21.15.77:0/278758766 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c011d20 con 0x7fd91815ca10 2023-07-04T18:51:00.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 -- 172.21.15.77:0/278758766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815ca10 msgr2=0x7fd918070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/278758766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815ca10 0x7fd918070870 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd9180705f0 tx=0x7fd90c009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.232 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 -- 172.21.15.77:0/278758766 shutdown_connections 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/278758766 >> v2:172.21.15.82:3300/0 conn(0x7fd918070db0 0x7fd9180cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/278758766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815ca10 0x7fd918070870 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 -- 172.21.15.77:0/278758766 >> 172.21.15.77:0/278758766 conn(0x7fd91806bba0 msgr2=0x7fd9180b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 -- 172.21.15.77:0/278758766 shutdown_connections 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.225+0000 7fd91ce5c700 1 -- 172.21.15.77:0/278758766 wait complete. 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.227+0000 7fd91ce5c700 1 Processor -- start 2023-07-04T18:51:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.227+0000 7fd91ce5c700 1 -- start start 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd918070db0 0x7fd9180c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd91815ca10 0x7fd9181e5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180b2850 con 0x7fd91815c640 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181e5d20 con 0x7fd918070db0 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91ce5c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181e5fc0 con 0x7fd91815ca10 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43328/0 (socket says 172.21.15.77:43328) 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 -- 172.21.15.77:0/3001477711 learned_addr learned my addr 172.21.15.77:0/3001477711 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd91659c700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd918070db0 0x7fd9180c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd916d9d700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd91815ca10 0x7fd9181e5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 -- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd91815ca10 msgr2=0x7fd9181e5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd91815ca10 0x7fd9181e5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 -- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd918070db0 msgr2=0x7fd9180c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd918070db0 0x7fd9180c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 -- 172.21.15.77:0/3001477711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c009550 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd915d9b700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c10 tx=0x7fd90c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.228+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c004510 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c01e070 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9181e62a0 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0038e0 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9180c9980 con 0x7fd91815c640 2023-07-04T18:51:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd90c003d40 con 0x7fd91815c640 2023-07-04T18:51:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd90c02dbd0 con 0x7fd91815c640 2023-07-04T18:51:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.229+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9180b2ce0 con 0x7fd91815c640 2023-07-04T18:51:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.232+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fd9180b2ce0 con 0x7fd91815c640 2023-07-04T18:51:00.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.302+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91804e210 con 0x7fd91815c640 2023-07-04T18:51:00.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.302+0000 7fd906ffd700 1 -- 172.21.15.77:0/3001477711 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd90c020070 con 0x7fd91815c640 2023-07-04T18:51:00.304 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:00.305 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:00.022059+0000","last_up_change":"0.000000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:00.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 msgr2=0x7fd918154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009c10 tx=0x7fd90c004750 comp rx=0 tx=0).stop 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 shutdown_connections 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd91815ca10 0x7fd9181e5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd918070db0 0x7fd9180c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 --2- 172.21.15.77:0/3001477711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91815c640 0x7fd918154230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 >> 172.21.15.77:0/3001477711 conn(0x7fd91806bba0 msgr2=0x7fd9180ca470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:00.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 shutdown_connections 2023-07-04T18:51:00.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:00.303+0000 7fd91ce5c700 1 -- 172.21.15.77:0/3001477711 wait complete. 2023-07-04T18:51:00.317 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-07-04T18:51:00.317 INFO:tasks.ceph:Waiting for OSDs to come up 2023-07-04T18:51:00.658 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T18:51:00.656+0000 7f4f75d18640 -1 Falling back to public interface 2023-07-04T18:51:00.666 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T18:51:00.664+0000 7f7e073ed640 -1 Falling back to public interface 2023-07-04T18:51:00.672 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:00.670+0000 7f4aad45a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-04T18:51:00.672 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T18:51:00.671+0000 7f2bb74d1640 -1 Falling back to public interface 2023-07-04T18:51:00.678 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T18:51:00.676+0000 7efd16c45640 -1 Falling back to public interface 2023-07-04T18:51:00.691 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T18:51:00.689+0000 7f2196524640 -1 Falling back to public interface 2023-07-04T18:51:00.692 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T18:51:00.688+0000 7f9b51a95640 -1 Falling back to public interface 2023-07-04T18:51:00.698 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T18:51:00.696+0000 7f0e79e11640 -1 Falling back to public interface 2023-07-04T18:51:00.704 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T18:51:00.702+0000 7fb43e403640 -1 Falling back to public interface 2023-07-04T18:51:00.752 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:00.750+0000 7f4aad45a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-04T18:51:00.890 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:00.888+0000 7f4aad45a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-04T18:51:00.920 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:00.919+0000 7f84ce4e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-04T18:51:00.999 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:00.998+0000 7f84ce4e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-04T18:51:01.125 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:01.126 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.125+0000 7f4aad45a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-04T18:51:01.137 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:01.136+0000 7f84ce4e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-04T18:51:01.200 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.198+0000 7f4aad45a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-04T18:51:01.237 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.236+0000 7fcceebca700 1 Processor -- start 2023-07-04T18:51:01.238 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.236+0000 7fcceebca700 1 -- start start 2023-07-04T18:51:01.240 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce80ce5a0 0x7fcce80ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.240 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcce806ee70 0x7fcce806d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.240 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcce806d7a0 0x7fcce80cb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80ceeb0 con 0x7fcce806ee70 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80cd630 con 0x7fcce806d7a0 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcceebca700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce806ddd0 con 0x7fcce80ce5a0 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcce806ee70 0x7fcce806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcce806ee70 0x7fcce806d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43334/0 (socket says 172.21.15.77:43334) 2023-07-04T18:51:01.241 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 -- 172.21.15.77:0/332133456 learned_addr learned my addr 172.21.15.77:0/332133456 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcced167700 1 --2- 172.21.15.77:0/332133456 >> v2:172.21.15.82:3300/0 conn(0x7fcce806d7a0 0x7fcce80cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fccec966700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce80ce5a0 0x7fcce80ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 -- 172.21.15.77:0/332133456 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce80ce5a0 msgr2=0x7fcce80ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce80ce5a0 0x7fcce80ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 -- 172.21.15.77:0/332133456 >> v2:172.21.15.82:3300/0 conn(0x7fcce806d7a0 msgr2=0x7fcce80cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:01.242 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 --2- 172.21.15.77:0/332133456 >> v2:172.21.15.82:3300/0 conn(0x7fcce806d7a0 0x7fcce80cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 -- 172.21.15.77:0/332133456 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce806e0b0 con 0x7fcce806ee70 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.237+0000 7fcce7fff700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcce8071620 tx=0x7fccd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5302af385e71eb94 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcce77fe700 1 -- 172.21.15.77:0/332133456 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4005e30 con 0x7fcce806ee70 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcce77fe700 1 -- 172.21.15.77:0/332133456 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4004030 con 0x7fcce806ee70 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcce77fe700 1 -- 172.21.15.77:0/332133456 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40043e0 con 0x7fcce806ee70 2023-07-04T18:51:01.243 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 -- 172.21.15.77:0/332133456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 msgr2=0x7fcce806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:01.244 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce806d260 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcce8071620 tx=0x7fccd4009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.244 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 -- 172.21.15.77:0/332133456 shutdown_connections 2023-07-04T18:51:01.244 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 --2- 172.21.15.77:0/332133456 >> v2:172.21.15.82:3300/0 conn(0x7fcce806d7a0 0x7fcce80cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.244 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce80ce5a0 0x7fcce80ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.245 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 --2- 172.21.15.77:0/332133456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce806d260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.245 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 -- 172.21.15.77:0/332133456 >> 172.21.15.77:0/332133456 conn(0x7fcce80664f0 msgr2=0x7fcce8099d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:01.245 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 -- 172.21.15.77:0/332133456 shutdown_connections 2023-07-04T18:51:01.245 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.238+0000 7fcceebca700 1 -- 172.21.15.77:0/332133456 wait complete. 2023-07-04T18:51:01.247 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.240+0000 7fcceebca700 1 Processor -- start 2023-07-04T18:51:01.248 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.240+0000 7fcceebca700 1 -- start start 2023-07-04T18:51:01.248 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce806d7a0 0x7fcce809c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.248 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.248 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcce80ce5a0 0x7fcce8159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:01.248 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80b4210 con 0x7fcce806ee70 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80b43f0 con 0x7fcce80ce5a0 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcceebca700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80b46d0 con 0x7fcce806d7a0 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43340/0 (socket says 172.21.15.77:43340) 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 -- 172.21.15.77:0/1644572892 learned_addr learned my addr 172.21.15.77:0/1644572892 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fccec966700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce806d7a0 0x7fcce809c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcced167700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcce80ce5a0 0x7fcce8159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:01.249 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 -- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce806d7a0 msgr2=0x7fcce809c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce806d7a0 0x7fcce809c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 -- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcce80ce5a0 msgr2=0x7fcce8159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcce80ce5a0 0x7fcce8159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.241+0000 7fcce7fff700 1 -- 172.21.15.77:0/1644572892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009500 con 0x7fcce806ee70 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce7fff700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009c10 tx=0x7fccd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4003bb0 con 0x7fcce806ee70 2023-07-04T18:51:01.250 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd4003d10 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce80b49b0 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd401c660 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce80b4f70 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fccd401cac0 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.242+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fccd4011af0 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.243+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce804e210 con 0x7fcce806ee70 2023-07-04T18:51:01.251 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.245+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fccd402a060 con 0x7fcce806ee70 2023-07-04T18:51:01.331 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.329+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce80b0430 con 0x7fcce806ee70 2023-07-04T18:51:01.331 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.330+0000 7fcce57fa700 1 -- 172.21.15.77:0/1644572892 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fccd4034090 con 0x7fcce806ee70 2023-07-04T18:51:01.332 INFO:teuthology.misc.health.smithi077.stdout: 2023-07-04T18:51:01.332 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:00.022059+0000","last_up_change":"0.000000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:01.333 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 msgr2=0x7fcce8158f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:01.333 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009c10 tx=0x7fccd4004750 comp rx=0 tx=0).stop 2023-07-04T18:51:01.333 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 shutdown_connections 2023-07-04T18:51:01.333 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcce80ce5a0 0x7fcce8159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.333 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcce806d7a0 0x7fcce809c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.334 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 --2- 172.21.15.77:0/1644572892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcce806ee70 0x7fcce8158f10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:01.334 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 >> 172.21.15.77:0/1644572892 conn(0x7fcce80664f0 msgr2=0x7fcce80cc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:01.334 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.331+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 shutdown_connections 2023-07-04T18:51:01.334 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:01.332+0000 7fcceebca700 1 -- 172.21.15.77:0/1644572892 wait complete. 2023-07-04T18:51:01.348 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-07-04T18:51:01.397 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:01.396+0000 7f84ce4e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-04T18:51:01.471 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.469+0000 7f4aad45a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-04T18:51:01.472 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:01.470+0000 7f84ce4e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-04T18:51:01.689 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.687+0000 7f4aad45a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-04T18:51:01.743 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:01.741+0000 7f84ce4e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-04T18:51:01.846 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.844+0000 7f4aad45a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-04T18:51:01.960 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:01.958+0000 7f84ce4e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-04T18:51:01.961 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:01.959+0000 7f4aad45a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-04T18:51:02.109 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:02.108+0000 7f4aad45a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-04T18:51:02.115 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:02.113+0000 7f84ce4e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-04T18:51:02.228 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:02.226+0000 7f84ce4e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-04T18:51:02.376 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:02.374+0000 7f84ce4e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-04T18:51:02.382 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:02.380+0000 7f4aad45a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-04T18:51:02.565 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:02.563+0000 7f4aad45a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-04T18:51:02.650 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:02.648+0000 7f84ce4e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-04T18:51:02.831 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:02.830+0000 7f84ce4e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-04T18:51:03.151 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:03.149+0000 7f4aad45a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-04T18:51:03.433 INFO:tasks.ceph.mgr.y.smithi077.stderr:2023-07-04T18:51:03.430+0000 7f84ce4e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-04T18:51:04.177 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:04.175+0000 7f4a656c1700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-04T18:51:04.177 INFO:tasks.ceph.mgr.x.smithi082.stderr:2023-07-04T18:51:04.175+0000 7f4a656c1700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-04T18:51:04.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T18:51:04.211+0000 7f4f75d18640 -1 osd.1 0 log_to_monitors true 2023-07-04T18:51:04.223 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T18:51:04.221+0000 7f0e79e11640 -1 osd.5 0 log_to_monitors true 2023-07-04T18:51:04.245 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T18:51:04.243+0000 7fb43e403640 -1 osd.7 0 log_to_monitors true 2023-07-04T18:51:04.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T18:51:04.246+0000 7f7e073ed640 -1 osd.0 0 log_to_monitors true 2023-07-04T18:51:04.250 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T18:51:04.248+0000 7f2196524640 -1 osd.6 0 log_to_monitors true 2023-07-04T18:51:04.251 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T18:51:04.249+0000 7efd16c45640 -1 osd.4 0 log_to_monitors true 2023-07-04T18:51:04.269 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T18:51:04.267+0000 7f2bb74d1640 -1 osd.3 0 log_to_monitors true 2023-07-04T18:51:04.272 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T18:51:04.269+0000 7f9b51a95640 -1 osd.2 0 log_to_monitors true 2023-07-04T18:51:08.126 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T18:51:08.123+0000 7efd0a7fc700 -1 osd.4 0 waiting for initial osdmap 2023-07-04T18:51:08.130 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T18:51:08.128+0000 7f7dfafa4700 -1 osd.0 0 waiting for initial osdmap 2023-07-04T18:51:08.136 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T18:51:08.135+0000 7efd02bd7700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.146 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T18:51:08.145+0000 7f7df337f700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.155 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:08.165 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T18:51:08.163+0000 7f0e6d9c8700 -1 osd.5 0 waiting for initial osdmap 2023-07-04T18:51:08.172 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T18:51:08.167+0000 7f2bab088700 -1 osd.3 0 waiting for initial osdmap 2023-07-04T18:51:08.178 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T18:51:08.176+0000 7f0e65da3700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T18:51:08.183+0000 7f2ba3463700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.220 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T18:51:08.215+0000 7f9b4564c700 -1 osd.2 0 waiting for initial osdmap 2023-07-04T18:51:08.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T18:51:08.218+0000 7f4f698cf700 -1 osd.1 0 waiting for initial osdmap 2023-07-04T18:51:08.231 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T18:51:08.230+0000 7f9b3da27700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T18:51:08.234+0000 7f4f61caa700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.238 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T18:51:08.237+0000 7f218a0db700 -1 osd.6 0 waiting for initial osdmap 2023-07-04T18:51:08.252 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T18:51:08.251+0000 7f21824b6700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.256 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T18:51:08.255+0000 7fb431fba700 -1 osd.7 0 waiting for initial osdmap 2023-07-04T18:51:08.266 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T18:51:08.265+0000 7fb42a395700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-04T18:51:08.278 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.276+0000 7f0cfe7f3700 1 Processor -- start 2023-07-04T18:51:08.278 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.276+0000 7f0cfe7f3700 1 -- start start 2023-07-04T18:51:08.280 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cfe7f3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cf80ae230 0x7f0cf815c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cfe7f3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf806a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cfe7f3700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf80af060 con 0x7f0cf806cfb0 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cfe7f3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf8070580 con 0x7f0cf80ae230 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cfe7f3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf815cf30 con 0x7f0cf80ac8a0 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cf7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cf80ae230 0x7f0cf815c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.281 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cf7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0cf80ae230 0x7f0cf815c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43396/0 (socket says 172.21.15.77:43396) 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.277+0000 7f0cf7fff700 1 -- 172.21.15.77:0/2963642621 learned_addr learned my addr 172.21.15.77:0/2963642621 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf77fe700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf806a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf7fff700 1 -- 172.21.15.77:0/2963642621 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 msgr2=0x7f0cf806a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf7fff700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf806a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf7fff700 1 -- 172.21.15.77:0/2963642621 >> v1:172.21.15.82:6789/0 conn(0x7f0cf806cfb0 legacy=0x7f0cf806d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf7fff700 1 -- 172.21.15.77:0/2963642621 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf815d210 con 0x7f0cf80ae230 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf7fff700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf80ae230 0x7f0cf815c950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80098f0 tx=0x7f0ce8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57e4958b60e207 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:08.282 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf6ffd700 1 -- 172.21.15.77:0/2963642621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8004230 con 0x7f0cf80ae230 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf6ffd700 1 -- 172.21.15.77:0/2963642621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce8004390 con 0x7f0cf80ae230 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cf6ffd700 1 -- 172.21.15.77:0/2963642621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80036a0 con 0x7f0cf80ae230 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/2963642621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf80ae230 msgr2=0x7f0cf815c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.278+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf80ae230 0x7f0cf815c950 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80098f0 tx=0x7f0ce8009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/2963642621 shutdown_connections 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf80ae230 0x7f0cf815c950 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.283 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/2963642621 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf806a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.284 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/2963642621 >> 172.21.15.77:0/2963642621 conn(0x7f0cf806af50 msgr2=0x7f0cf806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:08.284 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/2963642621 shutdown_connections 2023-07-04T18:51:08.284 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.279+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/2963642621 wait complete. 2023-07-04T18:51:08.286 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.280+0000 7f0cfe7f3700 1 Processor -- start 2023-07-04T18:51:08.286 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.280+0000 7f0cfe7f3700 1 -- start start 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf80ae230 0x7f0cf80b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf814efd0 con 0x7f0cf806cfb0 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81e7540 con 0x7f0cf80ae230 2023-07-04T18:51:08.287 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81e7820 con 0x7f0cf80ac8a0 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43400/0 (socket says 172.21.15.77:43400) 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 -- 172.21.15.77:0/429452077 learned_addr learned my addr 172.21.15.77:0/429452077 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf77fe700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfcd90700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf80ae230 0x7f0cf80b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.288 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 -- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 msgr2=0x7f0cf815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:08.289 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.289 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 -- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf80ae230 msgr2=0x7f0cf80b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:08.289 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf80ae230 0x7f0cf80b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.289 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 -- 172.21.15.77:0/429452077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009550 con 0x7f0cf806cfb0 2023-07-04T18:51:08.289 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf7fff700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009130 tx=0x7f0ce8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800b660 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800b7c0 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf81e7b00 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800bb70 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.281+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf81e80c0 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.282+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf80cc1d0 con 0x7f0cf806cfb0 2023-07-04T18:51:08.290 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.283+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 88981+0+0 (secure 0 0 0) 0x7f0ce800bcd0 con 0x7f0cf806cfb0 2023-07-04T18:51:08.291 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.283+0000 7f0cf4ff9700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 0x7f0cd806d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:08.291 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.283+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f0ce8089c50 con 0x7f0cf806cfb0 2023-07-04T18:51:08.292 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.283+0000 7f0cf77fe700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 0x7f0cd806d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:08.292 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.284+0000 7f0cf77fe700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 0x7f0cd806d100 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cf815bb70 tx=0x7f0ce0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:08.293 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.288+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ce8055d60 con 0x7f0cf806cfb0 2023-07-04T18:51:08.480 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.476+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf80cdd30 con 0x7f0cf806cfb0 2023-07-04T18:51:08.481 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.477+0000 7f0cf4ff9700 1 -- 172.21.15.77:0/429452077 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0ce801f090 con 0x7f0cf806cfb0 2023-07-04T18:51:08.481 INFO:teuthology.misc.health.smithi077.stdout: 2023-07-04T18:51:08.482 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":12,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:06.171555+0000","last_up_change":"0.000000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:08.483 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 msgr2=0x7f0cd806d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:08.484 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 0x7f0cd806d100 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cf815bb70 tx=0x7f0ce0008040 comp rx=0 tx=0).stop 2023-07-04T18:51:08.484 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 msgr2=0x7f0cf815eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:08.484 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009130 tx=0x7f0ce8004750 comp rx=0 tx=0).stop 2023-07-04T18:51:08.484 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 shutdown_connections 2023-07-04T18:51:08.484 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0cd806ac50 0x7f0cd806d100 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf806cfb0 0x7f0cf815eb40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf80ae230 0x7f0cf80b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 --2- 172.21.15.77:0/429452077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0cf80ac8a0 0x7f0cf815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 >> 172.21.15.77:0/429452077 conn(0x7f0cf806af50 msgr2=0x7f0cf806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 shutdown_connections 2023-07-04T18:51:08.485 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:08.479+0000 7f0cfe7f3700 1 -- 172.21.15.77:0/429452077 wait complete. 2023-07-04T18:51:08.495 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-07-04T18:51:15.302 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:15.428 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 Processor -- start 2023-07-04T18:51:15.429 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 -- start start 2023-07-04T18:51:15.429 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02380cb750 0x7f023806b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.429 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 0x7f023815d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.429 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 0x7f02380cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0238072150 con 0x7f02380cb750 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023806b7f0 con 0x7f02380ccdf0 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f023d197700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380cbb20 con 0x7f023815d240 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.424+0000 7f0236d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02380cb750 0x7f023806b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02380cb750 0x7f023806b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43584/0 (socket says 172.21.15.77:43584) 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 -- 172.21.15.77:0/1541360027 learned_addr learned my addr 172.21.15.77:0/1541360027 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f023759e700 1 --2- 172.21.15.77:0/1541360027 >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 0x7f02380cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.430 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f023659c700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 0x7f023815d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 -- 172.21.15.77:0/1541360027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 msgr2=0x7f023815d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 0x7f023815d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 -- 172.21.15.77:0/1541360027 >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 msgr2=0x7f02380cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1541360027 >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 0x7f02380cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 -- 172.21.15.77:0/1541360027 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02380701c0 con 0x7f02380cb750 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f023759e700 1 --2- 172.21.15.77:0/1541360027 >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 0x7f02380cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:15.431 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f023659c700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 0x7f023815d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380cb750 0x7f023806b2b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f023815e2e0 tx=0x7f022800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1b1b856aeba185 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0235d9b700 1 -- 172.21.15.77:0/1541360027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022800dd10 con 0x7f02380cb750 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0235d9b700 1 -- 172.21.15.77:0/1541360027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022800de70 con 0x7f02380cb750 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.425+0000 7f0235d9b700 1 -- 172.21.15.77:0/1541360027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228010670 con 0x7f02380cb750 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 -- 172.21.15.77:0/1541360027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380cb750 msgr2=0x7f023806b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380cb750 0x7f023806b2b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f023815e2e0 tx=0x7f022800bea0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.432 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 -- 172.21.15.77:0/1541360027 shutdown_connections 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 --2- 172.21.15.77:0/1541360027 >> v2:172.21.15.82:3300/0 conn(0x7f02380ccdf0 0x7f02380cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380cb750 0x7f023806b2b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 --2- 172.21.15.77:0/1541360027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f023815d240 0x7f023815d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 -- 172.21.15.77:0/1541360027 >> 172.21.15.77:0/1541360027 conn(0x7f02380cf220 msgr2=0x7f02380cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 -- 172.21.15.77:0/1541360027 shutdown_connections 2023-07-04T18:51:15.433 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.426+0000 7f023d197700 1 -- 172.21.15.77:0/1541360027 wait complete. 2023-07-04T18:51:15.436 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.428+0000 7f023d197700 1 Processor -- start 2023-07-04T18:51:15.436 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.428+0000 7f023d197700 1 -- start start 2023-07-04T18:51:15.436 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f02380cb750 0x7f02380c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.436 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 0x7f02380c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023809f690 con 0x7f02380ccdf0 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02380b3a70 con 0x7f023815d240 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023d197700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02380b3cc0 con 0x7f02380cb750 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43590/0 (socket says 172.21.15.77:43590) 2023-07-04T18:51:15.437 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 -- 172.21.15.77:0/1108879981 learned_addr learned my addr 172.21.15.77:0/1108879981 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f02380cb750 0x7f02380c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023759e700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 0x7f02380c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 -- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f02380cb750 msgr2=0x7f02380c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f02380cb750 0x7f02380c4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 -- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 msgr2=0x7f02380c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 0x7f02380c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.438 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 -- 172.21.15.77:0/1108879981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022800b7f0 con 0x7f02380ccdf0 2023-07-04T18:51:15.439 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023759e700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 0x7f02380c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:15.439 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f023659c700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c7f0 tx=0x7f022c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.439 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.429+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c011800 con 0x7f02380ccdf0 2023-07-04T18:51:15.439 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.430+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c011960 con 0x7f02380ccdf0 2023-07-04T18:51:15.439 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.430+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02380b3fa0 con 0x7f02380ccdf0 2023-07-04T18:51:15.440 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.430+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c011d10 con 0x7f02380ccdf0 2023-07-04T18:51:15.440 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.430+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02380b4560 con 0x7f02380ccdf0 2023-07-04T18:51:15.441 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.431+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023804e210 con 0x7f02380ccdf0 2023-07-04T18:51:15.447 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.431+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f022c018360 con 0x7f02380ccdf0 2023-07-04T18:51:15.448 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.431+0000 7f021f7fe700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 0x7f022406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.448 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.431+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f022c088fe0 con 0x7f02380ccdf0 2023-07-04T18:51:15.448 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.431+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 0x7f022406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.448 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.432+0000 7f0236d9d700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 0x7f022406d190 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0228000c00 tx=0x7f0228013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.449 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.436+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f022c054540 con 0x7f02380ccdf0 2023-07-04T18:51:15.625 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.623+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0238159b90 con 0x7f02380ccdf0 2023-07-04T18:51:15.626 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.624+0000 7f021f7fe700 1 -- 172.21.15.77:0/1108879981 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12814 (secure 0 0 0) 0x7f022c057b60 con 0x7f02380ccdf0 2023-07-04T18:51:15.626 INFO:teuthology.misc.health.smithi077.stdout: 2023-07-04T18:51:15.627 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":16,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:12.140098+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:15.629 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 msgr2=0x7f022406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.630 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 0x7f022406d190 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0228000c00 tx=0x7f0228013000 comp rx=0 tx=0).stop 2023-07-04T18:51:15.630 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 msgr2=0x7f02380c5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.631 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c7f0 tx=0x7f022c00cbb0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.631 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 shutdown_connections 2023-07-04T18:51:15.631 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022406ace0 0x7f022406d190 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.632 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f023815d240 0x7f02380c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.632 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f02380cb750 0x7f02380c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.632 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 --2- 172.21.15.77:0/1108879981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02380ccdf0 0x7f02380c5080 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.633 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 >> 172.21.15.77:0/1108879981 conn(0x7f02380cf220 msgr2=0x7f023806c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:15.633 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 shutdown_connections 2023-07-04T18:51:15.633 INFO:teuthology.misc.health.smithi077.stderr:2023-07-04T18:51:15.626+0000 7f023d197700 1 -- 172.21.15.77:0/1108879981 wait complete. 2023-07-04T18:51:15.639 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-07-04T18:51:15.639 INFO:tasks.ceph:Creating RBD pool 2023-07-04T18:51:15.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool create rbd 8 2023-07-04T18:51:15.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.758+0000 7f1af1785700 1 Processor -- start 2023-07-04T18:51:15.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.758+0000 7f1af1785700 1 -- start start 2023-07-04T18:51:15.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1af1785700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f1aec0d08e0 0x7f1aec0d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1af1785700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aec0cde40 0x7f1aec0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1af1785700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec0cd9b0 con 0x7f1aec15fca0 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1af1785700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0d12c0 con 0x7f1aec0cde40 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1af1785700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec160070 con 0x7f1aec0d08e0 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aec0cde40 0x7f1aec0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1aec0cde40 0x7f1aec0ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43600/0 (socket says 172.21.15.77:43600) 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 -- 172.21.15.77:0/66076889 learned_addr learned my addr 172.21.15.77:0/66076889 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aea7fc700 1 --2- 172.21.15.77:0/66076889 >> v2:172.21.15.82:3300/0 conn(0x7f1aec0d08e0 0x7f1aec0d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 -- 172.21.15.77:0/66076889 >> v2:172.21.15.82:3300/0 conn(0x7f1aec0d08e0 msgr2=0x7f1aec0d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 --2- 172.21.15.77:0/66076889 >> v2:172.21.15.82:3300/0 conn(0x7f1aec0d08e0 0x7f1aec0d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 -- 172.21.15.77:0/66076889 >> v1:172.21.15.82:6789/0 conn(0x7f1aec15fca0 legacy=0x7f1aec160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.759+0000 7f1aeb7fe700 1 -- 172.21.15.77:0/66076889 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec160d50 con 0x7f1aec0cde40 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1aeb7fe700 1 --2- 172.21.15.77:0/66076889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec0ce280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1aec06a260 tx=0x7f1ae400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e39d5e8f52ec311 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1ae9ffb700 1 -- 172.21.15.77:0/66076889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae400e070 con 0x7f1aec0cde40 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1ae9ffb700 1 -- 172.21.15.77:0/66076889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae400b910 con 0x7f1aec0cde40 2023-07-04T18:51:15.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1ae9ffb700 1 -- 172.21.15.77:0/66076889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae400bcc0 con 0x7f1aec0cde40 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 -- 172.21.15.77:0/66076889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 msgr2=0x7f1aec0ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 --2- 172.21.15.77:0/66076889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec0ce280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1aec06a260 tx=0x7f1ae400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 -- 172.21.15.77:0/66076889 shutdown_connections 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 --2- 172.21.15.77:0/66076889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec0ce280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 --2- 172.21.15.77:0/66076889 >> v2:172.21.15.82:3300/0 conn(0x7f1aec0d08e0 0x7f1aec0d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 -- 172.21.15.77:0/66076889 >> 172.21.15.77:0/66076889 conn(0x7f1aec071e90 msgr2=0x7f1aec09bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:15.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 -- 172.21.15.77:0/66076889 shutdown_connections 2023-07-04T18:51:15.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.760+0000 7f1af1785700 1 -- 172.21.15.77:0/66076889 wait complete. 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 Processor -- start 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 -- start start 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1aec0d08e0 0x7f1aec0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 0x7f1aec0b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0b82b0 con 0x7f1aec0cde40 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0b76a0 con 0x7f1aec15fca0 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1af1785700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0b7980 con 0x7f1aec0d08e0 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1aeaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.762+0000 7f1aeaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43602/0 (socket says 172.21.15.77:43602) 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 -- 172.21.15.77:0/1055015347 learned_addr learned my addr 172.21.15.77:0/1055015347 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeb7fe700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 0x7f1aec0b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aea7fc700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1aec0d08e0 0x7f1aec0b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 -- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1aec0d08e0 msgr2=0x7f1aec0b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1aec0d08e0 0x7f1aec0b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 -- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 msgr2=0x7f1aec0b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 0x7f1aec0b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 -- 172.21.15.77:0/1055015347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4009b30 con 0x7f1aec0cde40 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeb7fe700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 0x7f1aec0b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1aeaffd700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000c450 tx=0x7f1ae000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00052d0 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0005430 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec0b7c60 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00038c0 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.763+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0b72b0 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.764+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec0b7440 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.764+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1ae0003a20 con 0x7f1aec0cde40 2023-07-04T18:51:15.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.765+0000 7f1ad37fe700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 0x7f1ad806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.765+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ae00882e0 con 0x7f1aec0cde40 2023-07-04T18:51:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.765+0000 7f1aea7fc700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 0x7f1ad806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.766+0000 7f1aea7fc700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 0x7f1ad806d190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0c6f60 tx=0x7f1adc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:15.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.769+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1ae00538d0 con 0x7f1aec0cde40 2023-07-04T18:51:15.958 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:15.956+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f1aec04e210 con 0x7f1aec0cde40 2023-07-04T18:51:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.307+0000 7f1ad37fe700 1 -- 172.21.15.77:0/1055015347 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f1ae0056ef0 con 0x7f1aec0cde40 2023-07-04T18:51:16.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 msgr2=0x7f1ad806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 0x7f1ad806d190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0c6f60 tx=0x7f1adc008040 comp rx=0 tx=0).stop 2023-07-04T18:51:16.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 msgr2=0x7f1aec153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000c450 tx=0x7f1ae000c810 comp rx=0 tx=0).stop 2023-07-04T18:51:16.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 shutdown_connections 2023-07-04T18:51:16.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1ad806ace0 0x7f1ad806d190 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0c6f60 tx=0x7f1adc008040 comp rx=0 tx=0).stop 2023-07-04T18:51:16.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aec0cde40 0x7f1aec153790 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1aec15fca0 0x7f1aec0b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 --2- 172.21.15.77:0/1055015347 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1aec0d08e0 0x7f1aec0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 >> 172.21.15.77:0/1055015347 conn(0x7f1aec071e90 msgr2=0x7f1aec0afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 shutdown_connections 2023-07-04T18:51:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.310+0000 7f1af1785700 1 -- 172.21.15.77:0/1055015347 wait complete. 2023-07-04T18:51:16.316 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2023-07-04T18:51:16.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-07-04T18:51:16.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.447+0000 7f2aa5f8d700 1 Processor -- start 2023-07-04T18:51:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.447+0000 7f2aa5f8d700 1 -- start start 2023-07-04T18:51:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2aa5f8d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aa00b3f10 0x7f2aa00ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2aa5f8d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa015e810 con 0x7f2aa00ce420 2023-07-04T18:51:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2aa5f8d700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00ce920 con 0x7f2aa015a430 2023-07-04T18:51:16.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2aa5f8d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa015e1c0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2a9effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aa00b3f10 0x7f2aa00ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2a9ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2aa00ce420 0x7f2aa00cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50752/0 (socket says 172.21.15.77:50752) 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2a9ffff700 1 -- 172.21.15.77:0/1134293546 learned_addr learned my addr 172.21.15.77:0/1134293546 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.448+0000 7f2a9effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2aa00b3f10 0x7f2aa00ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43612/0 (socket says 172.21.15.77:43612) 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9effd700 1 -- 172.21.15.77:0/1134293546 >> v1:172.21.15.82:6789/0 conn(0x7f2aa015a430 legacy=0x7f2aa00b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9effd700 1 -- 172.21.15.77:0/1134293546 >> v1:172.21.15.77:6789/0 conn(0x7f2aa00ce420 legacy=0x7f2aa00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9effd700 1 -- 172.21.15.77:0/1134293546 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa015e4a0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9e7fc700 1 -- 172.21.15.77:0/1134293546 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083511821 0 0) 0x7f2aa015e810 con 0x7f2aa00ce420 2023-07-04T18:51:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9effd700 1 --2- 172.21.15.77:0/1134293546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa00ccc40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009840 tx=0x7f2a90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7f78182d4d9e71 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9e7fc700 1 -- 172.21.15.77:0/1134293546 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90011830 con 0x7f2aa00b3f10 2023-07-04T18:51:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9e7fc700 1 -- 172.21.15.77:0/1134293546 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90011990 con 0x7f2aa00b3f10 2023-07-04T18:51:16.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.449+0000 7f2a9e7fc700 1 -- 172.21.15.77:0/1134293546 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90011d20 con 0x7f2aa00b3f10 2023-07-04T18:51:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/1134293546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 msgr2=0x7f2aa00ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/1134293546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa00ccc40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009840 tx=0x7f2a90009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/1134293546 shutdown_connections 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/1134293546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa00ccc40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/1134293546 >> 172.21.15.77:0/1134293546 conn(0x7f2aa00cf180 msgr2=0x7f2aa00cf550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/1134293546 shutdown_connections 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.450+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/1134293546 wait complete. 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.451+0000 7f2aa5f8d700 1 Processor -- start 2023-07-04T18:51:16.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.451+0000 7f2aa5f8d700 1 -- start start 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 0x7f2aa0151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2aa015a430 0x7f2aa00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0071d70 con 0x7f2aa00b3f10 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00caa50 con 0x7f2aa015a430 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015b6e0 con 0x7f2aa00ce420 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:16.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43616/0 (socket says 172.21.15.77:43616) 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 -- 172.21.15.77:0/3817938385 learned_addr learned my addr 172.21.15.77:0/3817938385 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9ffff700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2aa015a430 0x7f2aa00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9effd700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 0x7f2aa0151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 -- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 msgr2=0x7f2aa0151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 0x7f2aa0151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 -- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2aa015a430 msgr2=0x7f2aa00b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:16.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2aa015a430 0x7f2aa00b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 -- 172.21.15.77:0/3817938385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a900094a0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9effd700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 0x7f2aa0151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a9f7fe700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c450 tx=0x7f2a9400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a940052d0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94005430 con 0x7f2aa00b3f10 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa015b980 con 0x7f2aa00b3f10 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.452+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a940038c0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.453+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa015bf40 con 0x7f2aa00b3f10 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.453+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa004e210 con 0x7f2aa00b3f10 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.454+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2a94003a20 con 0x7f2aa00b3f10 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.454+0000 7f2a87fff700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 0x7f2a8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.454+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f2a94088530 con 0x7f2aa00b3f10 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.454+0000 7f2a9effd700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 0x7f2a8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:16.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.455+0000 7f2a9effd700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 0x7f2a8c06d190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00b4300 tx=0x7f2a9001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:16.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.459+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2a940539d0 con 0x7f2aa00b3f10 2023-07-04T18:51:16.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:16.649+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2aa006b510 con 0x7f2aa00b3f10 2023-07-04T18:51:17.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.308+0000 7f2a87fff700 1 -- 172.21.15.77:0/3817938385 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2a94056ff0 con 0x7f2aa00b3f10 2023-07-04T18:51:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 msgr2=0x7f2a8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 0x7f2a8c06d190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00b4300 tx=0x7f2a9001c040 comp rx=0 tx=0).stop 2023-07-04T18:51:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 msgr2=0x7f2aa015c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c450 tx=0x7f2a9400c810 comp rx=0 tx=0).stop 2023-07-04T18:51:17.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 shutdown_connections 2023-07-04T18:51:17.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a8c06ace0 0x7f2a8c06d190 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2aa00b3f10 0x7f2aa015c740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2aa015a430 0x7f2aa00b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 --2- 172.21.15.77:0/3817938385 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2aa00ce420 0x7f2aa0151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.311+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 >> 172.21.15.77:0/3817938385 conn(0x7f2aa00cf180 msgr2=0x7f2aa00b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.313+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 shutdown_connections 2023-07-04T18:51:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.313+0000 7f2aa5f8d700 1 -- 172.21.15.77:0/3817938385 wait complete. 2023-07-04T18:51:17.315 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2023-07-04T18:51:17.328 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-07-04T18:51:17.329 INFO:tasks.ceph.mds.a:Restarting daemon 2023-07-04T18:51:17.329 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-07-04T18:51:17.331 INFO:tasks.ceph.mds.a:Started 2023-07-04T18:51:17.332 INFO:tasks.ceph.mds.c:Restarting daemon 2023-07-04T18:51:17.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-07-04T18:51:17.334 INFO:tasks.ceph.mds.c:Started 2023-07-04T18:51:17.334 INFO:tasks.ceph.mds.b:Restarting daemon 2023-07-04T18:51:17.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-07-04T18:51:17.337 INFO:tasks.ceph.mds.b:Started 2023-07-04T18:51:17.337 INFO:tasks.ceph.mds.d:Restarting daemon 2023-07-04T18:51:17.338 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-07-04T18:51:17.340 INFO:tasks.ceph.mds.d:Started 2023-07-04T18:51:17.340 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-07-04T18:51:17.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:17.411 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2023-07-04T18:51:17.412 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2023-07-04T18:51:17.422 INFO:tasks.ceph.mds.d.smithi082.stdout:starting mds.d at 2023-07-04T18:51:17.423 INFO:tasks.ceph.mds.b.smithi082.stdout:starting mds.b at 2023-07-04T18:51:17.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.469+0000 7f93f0fe0700 1 Processor -- start 2023-07-04T18:51:17.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.469+0000 7f93f0fe0700 1 -- start start 2023-07-04T18:51:17.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93f0fe0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15a430 0x7f93ec0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93f0fe0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93ec15ea40 0x7f93ec0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93f0fe0700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0cf100 con 0x7f93ec0cebf0 2023-07-04T18:51:17.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93f0fe0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec15df60 con 0x7f93ec15ea40 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93f0fe0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec15e240 con 0x7f93ec15a430 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93ead9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93ec15ea40 0x7f93ec0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93ead9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93ec15ea40 0x7f93ec0cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43654/0 (socket says 172.21.15.77:43654) 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.470+0000 7f93ead9d700 1 -- 172.21.15.77:0/2467112835 learned_addr learned my addr 172.21.15.77:0/2467112835 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ea59c700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15a430 0x7f93ec0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ead9d700 1 -- 172.21.15.77:0/2467112835 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15a430 msgr2=0x7f93ec0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ead9d700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15a430 0x7f93ec0ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ead9d700 1 -- 172.21.15.77:0/2467112835 >> v1:172.21.15.82:6789/0 conn(0x7f93ec0cebf0 legacy=0x7f93ec0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ead9d700 1 -- 172.21.15.77:0/2467112835 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec15e520 con 0x7f93ec15ea40 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93ead9d700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec15ea40 0x7f93ec0cfaf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009e50 tx=0x7f93e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0749fa22f68cd04 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93e959a700 1 -- 172.21.15.77:0/2467112835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e400e070 con 0x7f93ec15ea40 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93e959a700 1 -- 172.21.15.77:0/2467112835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e400b910 con 0x7f93ec15ea40 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93e959a700 1 -- 172.21.15.77:0/2467112835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e400bcc0 con 0x7f93ec15ea40 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2467112835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec15ea40 msgr2=0x7f93ec0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.471+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec15ea40 0x7f93ec0cfaf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009e50 tx=0x7f93e400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2467112835 shutdown_connections 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec15ea40 0x7f93ec0cfaf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2467112835 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15a430 0x7f93ec0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2467112835 >> 172.21.15.77:0/2467112835 conn(0x7f93ec06d7c0 msgr2=0x7f93ec06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2467112835 shutdown_connections 2023-07-04T18:51:17.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.472+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2467112835 wait complete. 2023-07-04T18:51:17.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.473+0000 7f93f0fe0700 1 Processor -- start 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.473+0000 7f93f0fe0700 1 -- start start 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15ea40 0x7f93ec0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93ec151df0 0x7f93ec0b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec1e4d40 con 0x7f93ec0cebf0 2023-07-04T18:51:17.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec1e4130 con 0x7f93ec151df0 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec1e4410 con 0x7f93ec15ea40 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43656/0 (socket says 172.21.15.77:43656) 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 -- 172.21.15.77:0/2298808669 learned_addr learned my addr 172.21.15.77:0/2298808669 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93e9d9b700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15ea40 0x7f93ec0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ead9d700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93ec151df0 0x7f93ec0b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 -- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15ea40 msgr2=0x7f93ec0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15ea40 0x7f93ec0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 -- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93ec151df0 msgr2=0x7f93ec0b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93ec151df0 0x7f93ec0b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 -- 172.21.15.77:0/2298808669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e4009b30 con 0x7f93ec0cebf0 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93ea59c700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93e000e9b0 tx=0x7f93e000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0009710 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e000ee50 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec1e46f0 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e000ca50 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.474+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec0b1de0 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.475+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04e210 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.476+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f93e000cbb0 con 0x7f93ec0cebf0 2023-07-04T18:51:17.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.476+0000 7f93daffd700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 0x7f93cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.476+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f93e0089100 con 0x7f93ec0cebf0 2023-07-04T18:51:17.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.476+0000 7f93e9d9b700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 0x7f93cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.477+0000 7f93e9d9b700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 0x7f93cc06d190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93ec159680 tx=0x7f93dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.481+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f93e0057f90 con 0x7f93ec0cebf0 2023-07-04T18:51:17.671 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.669+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93ec0ce270 con 0x7f93ec0cebf0 2023-07-04T18:51:17.672 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.670+0000 7f93daffd700 1 -- 172.21.15.77:0/2298808669 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14613 (secure 0 0 0) 0x7f93e000ce60 con 0x7f93ec0cebf0 2023-07-04T18:51:17.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:17.673 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":18,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:17.306123+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:17.674 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 msgr2=0x7f93cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.674 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 0x7f93cc06d190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f93ec159680 tx=0x7f93dc009040 comp rx=0 tx=0).stop 2023-07-04T18:51:17.674 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 msgr2=0x7f93ec1e3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.674 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f93e000e9b0 tx=0x7f93e000ecc0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.675 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 shutdown_connections 2023-07-04T18:51:17.675 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f93cc06ace0 0x7f93cc06d190 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.675 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93ec0cebf0 0x7f93ec1e3b50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.675 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93ec151df0 0x7f93ec0b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 --2- 172.21.15.77:0/2298808669 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f93ec15ea40 0x7f93ec0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 >> 172.21.15.77:0/2298808669 conn(0x7f93ec06d7c0 msgr2=0x7f93ec154b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:17.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 shutdown_connections 2023-07-04T18:51:17.677 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.672+0000 7f93f0fe0700 1 -- 172.21.15.77:0/2298808669 wait complete. 2023-07-04T18:51:17.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:17.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:17.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.807+0000 7ff406658700 1 Processor -- start 2023-07-04T18:51:17.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.807+0000 7ff406658700 1 -- start start 2023-07-04T18:51:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff406658700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4000d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff406658700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff40015ae00 0x7ff40006ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff406658700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40006a880 con 0x7ff40015a9d0 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff406658700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4000d38a0 con 0x7ff40015ae00 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff406658700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000d12f0 con 0x7ff4000d09e0 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff3ff7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff40015a9d0 0x7ff4001600a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50802/0 (socket says 172.21.15.77:50802) 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff3ff7fe700 1 -- 172.21.15.77:0/2490524299 learned_addr learned my addr 172.21.15.77:0/2490524299 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2490524299 >> v2:172.21.15.77:3300/0 conn(0x7ff40015ae00 0x7ff40006ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff3fffff700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4000d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff3feffd700 1 -- 172.21.15.77:0/2490524299 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263581502 0 0) 0x7ff40006a880 con 0x7ff40015a9d0 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff3feffd700 1 -- 172.21.15.77:0/2490524299 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e80035b0 con 0x7ff40015a9d0 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff404bf5700 1 -- 172.21.15.77:0/2490524299 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 msgr2=0x7ff4000d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4000d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff404bf5700 1 -- 172.21.15.77:0/2490524299 >> v1:172.21.15.77:6789/0 conn(0x7ff40015a9d0 legacy=0x7ff4001600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.808+0000 7ff404bf5700 1 -- 172.21.15.77:0/2490524299 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff400160b40 con 0x7ff40015ae00 2023-07-04T18:51:17.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff40006ad10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009d90 tx=0x7ff3f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdc92ec76f44cc7 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff3feffd700 1 -- 172.21.15.77:0/2490524299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800e070 con 0x7ff40015ae00 2023-07-04T18:51:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff3feffd700 1 -- 172.21.15.77:0/2490524299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800bb20 con 0x7ff40015ae00 2023-07-04T18:51:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff3feffd700 1 -- 172.21.15.77:0/2490524299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8012430 con 0x7ff40015ae00 2023-07-04T18:51:17.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 -- 172.21.15.77:0/2490524299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 msgr2=0x7ff40006ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff40006ad10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009d90 tx=0x7ff3f800b330 comp rx=0 tx=0).stop 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 -- 172.21.15.77:0/2490524299 shutdown_connections 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff40006ad10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 --2- 172.21.15.77:0/2490524299 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4000d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 -- 172.21.15.77:0/2490524299 >> 172.21.15.77:0/2490524299 conn(0x7ff400071e90 msgr2=0x7ff40009bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 -- 172.21.15.77:0/2490524299 shutdown_connections 2023-07-04T18:51:17.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.809+0000 7ff406658700 1 -- 172.21.15.77:0/2490524299 wait complete. 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.810+0000 7ff406658700 1 Processor -- start 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 -- start start 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4001593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff40015a9d0 0x7ff4000cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4001e8770 con 0x7ff40015ae00 2023-07-04T18:51:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4001e7b60 con 0x7ff40015a9d0 2023-07-04T18:51:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.811+0000 7ff406658700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4001e7e40 con 0x7ff4000d09e0 2023-07-04T18:51:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43672/0 (socket says 172.21.15.77:43672) 2023-07-04T18:51:17.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 -- 172.21.15.77:0/2771905579 learned_addr learned my addr 172.21.15.77:0/2771905579 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff3ff7fe700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff40015a9d0 0x7ff4000cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff3fffff700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4001593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 -- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 msgr2=0x7ff4001593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4001593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 -- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff40015a9d0 msgr2=0x7ff4000cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff40015a9d0 0x7ff4000cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 -- 172.21.15.77:0/2771905579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8009a70 con 0x7ff40015ae00 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff404bf5700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800b370 tx=0x7ff3f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8012430 con 0x7ff40015ae00 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f8020070 con 0x7ff40015ae00 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80098f0 con 0x7ff40015ae00 2023-07-04T18:51:17.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4001e8120 con 0x7ff40015ae00 2023-07-04T18:51:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.812+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff400159920 con 0x7ff40015ae00 2023-07-04T18:51:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.813+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40004e210 con 0x7ff40015ae00 2023-07-04T18:51:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.813+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff3f8004040 con 0x7ff40015ae00 2023-07-04T18:51:17.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.814+0000 7ff3fcff9700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 0x7ff3ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.814+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff3f8022070 con 0x7ff40015ae00 2023-07-04T18:51:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.814+0000 7ff3fffff700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 0x7ff3ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.814+0000 7ff3fffff700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 0x7ff3ec06d190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009bc0 tx=0x7ff3f40091c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:17.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:17.818+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3f8059b20 con 0x7ff40015ae00 2023-07-04T18:51:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.001+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff40006bee0 con 0x7ff40015ae00 2023-07-04T18:51:18.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.002+0000 7ff3fcff9700 1 -- 172.21.15.77:0/2771905579 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3f802c020 con 0x7ff40015ae00 2023-07-04T18:51:18.004 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:18.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 msgr2=0x7ff3ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 0x7ff3ec06d190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009bc0 tx=0x7ff3f40091c0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 msgr2=0x7ff4000cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800b370 tx=0x7ff3f8004ad0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 shutdown_connections 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3ec06ace0 0x7ff3ec06d190 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40015ae00 0x7ff4000cf4f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4000d09e0 0x7ff4001593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 --2- 172.21.15.77:0/2771905579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff40015a9d0 0x7ff4000cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 >> 172.21.15.77:0/2771905579 conn(0x7ff400071e90 msgr2=0x7ff40015e310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 shutdown_connections 2023-07-04T18:51:18.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.004+0000 7ff406658700 1 -- 172.21.15.77:0/2771905579 wait complete. 2023-07-04T18:51:18.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:18.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.138+0000 7efec8271700 1 Processor -- start 2023-07-04T18:51:18.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.138+0000 7efec8271700 1 -- start start 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec8271700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7efec015d520 0x7efec0069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec8271700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec006eff0 con 0x7efec006ec20 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec8271700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00ac8a0 con 0x7efec006a380 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec8271700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec015d090 con 0x7efec015d520 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec600d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efec006ec20 0x7efec0160a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50814/0 (socket says 172.21.15.77:50814) 2023-07-04T18:51:18.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec600d700 1 -- 172.21.15.77:0/2309959917 learned_addr learned my addr 172.21.15.77:0/2309959917 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec580c700 1 --2- 172.21.15.77:0/2309959917 >> v2:172.21.15.82:3300/0 conn(0x7efec015d520 0x7efec0069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311934144 0 0) 0x7efec006eff0 con 0x7efec006ec20 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac0035b0 con 0x7efec006ec20 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105251727 0 0) 0x7efeac0035b0 con 0x7efec006ec20 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 >> v2:172.21.15.82:3300/0 conn(0x7efec015d520 msgr2=0x7efec0069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 --2- 172.21.15.77:0/2309959917 >> v2:172.21.15.82:3300/0 conn(0x7efec015d520 0x7efec0069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 >> v1:172.21.15.82:6789/0 conn(0x7efec006a380 legacy=0x7efec00b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec006a7c0 con 0x7efec006ec20 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7efebc001670 con 0x7efec006ec20 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 >> v1:172.21.15.77:6789/0 conn(0x7efec006ec20 legacy=0x7efec0160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efeac003a00 0x7efeac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.139+0000 7efec500b700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac004850 0x7efeac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 0x7efeac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec015d090 con 0x7efeac005200 2023-07-04T18:51:18.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac00c080 con 0x7efeac004850 2023-07-04T18:51:18.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeac00c240 con 0x7efeac003a00 2023-07-04T18:51:18.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 0x7efeac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec600d700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efeac003a00 0x7efeac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec580c700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac004850 0x7efeac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 -- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efeac003a00 msgr2=0x7efeac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efeac003a00 0x7efeac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 -- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac004850 msgr2=0x7efeac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac004850 0x7efeac004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 -- 172.21.15.77:0/2309959917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0035b0 con 0x7efeac005200 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 -- 172.21.15.77:0/2309959917 reap_dead start 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec680e700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 0x7efeac00ba10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c190 tx=0x7efeac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221160b7bd8481c9 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8001db0 con 0x7efeac005200 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb800ccf0 con 0x7efeac005200 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.140+0000 7efec500b700 1 -- 172.21.15.77:0/2309959917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8003c40 con 0x7efeac005200 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 -- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 msgr2=0x7efeac00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 0x7efeac00ba10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c190 tx=0x7efeac005880 comp rx=0 tx=0).stop 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 -- 172.21.15.77:0/2309959917 shutdown_connections 2023-07-04T18:51:18.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 --2- 172.21.15.77:0/2309959917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efeac005200 0x7efeac00ba10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 -- 172.21.15.77:0/2309959917 >> 172.21.15.77:0/2309959917 conn(0x7efec006af50 msgr2=0x7efec006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 -- 172.21.15.77:0/2309959917 shutdown_connections 2023-07-04T18:51:18.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.141+0000 7efec8271700 1 -- 172.21.15.77:0/2309959917 wait complete. 2023-07-04T18:51:18.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.142+0000 7efec8271700 1 Processor -- start 2023-07-04T18:51:18.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.142+0000 7efec8271700 1 -- start start 2023-07-04T18:51:18.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac005200 0x7efec0150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.152 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efec014bbb0 0x7efec0151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec014c870 con 0x7efec014c040 2023-07-04T18:51:18.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01521d0 con 0x7efeac005200 2023-07-04T18:51:18.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec0150bf0 con 0x7efec014bbb0 2023-07-04T18:51:18.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43692/0 (socket says 172.21.15.77:43692) 2023-07-04T18:51:18.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 -- 172.21.15.77:0/17861447 learned_addr learned my addr 172.21.15.77:0/17861447 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:18.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec600d700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac005200 0x7efec0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec580c700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efec014bbb0 0x7efec0151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 -- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efec014bbb0 msgr2=0x7efec0151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efec014bbb0 0x7efec0151c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 -- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac005200 msgr2=0x7efec0150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac005200 0x7efec0150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 -- 172.21.15.77:0/17861447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0150ed0 con 0x7efec014c040 2023-07-04T18:51:18.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec680e700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efeb8005040 tx=0x7efeb80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80052d0 con 0x7efec014c040 2023-07-04T18:51:18.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb8005430 con 0x7efec014c040 2023-07-04T18:51:18.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80057a0 con 0x7efec014c040 2023-07-04T18:51:18.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec01511b0 con 0x7efec014c040 2023-07-04T18:51:18.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.143+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec0151770 con 0x7efec014c040 2023-07-04T18:51:18.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.144+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec004e210 con 0x7efec014c040 2023-07-04T18:51:18.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.145+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efeb801d480 con 0x7efec014c040 2023-07-04T18:51:18.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.146+0000 7efeb6ffd700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 0x7efea406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.146+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efeb808a150 con 0x7efec014c040 2023-07-04T18:51:18.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.146+0000 7efec600d700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 0x7efea406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.146+0000 7efec600d700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 0x7efea406d190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efebc0011e0 tx=0x7efebc000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.160 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.149+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efeb8055950 con 0x7efec014c040 2023-07-04T18:51:18.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.341+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efec014cae0 con 0x7efec014c040 2023-07-04T18:51:18.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.341+0000 7efeb6ffd700 1 -- 172.21.15.77:0/17861447 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7efeb8058f70 con 0x7efec014c040 2023-07-04T18:51:18.344 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:18.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 msgr2=0x7efea406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 0x7efea406d190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efebc0011e0 tx=0x7efebc000bc0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 msgr2=0x7efec014f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efeb8005040 tx=0x7efeb80014a0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 shutdown_connections 2023-07-04T18:51:18.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efea406ace0 0x7efea406d190 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efec014c040 0x7efec014f1b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efeac005200 0x7efec0150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 --2- 172.21.15.77:0/17861447 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efec014bbb0 0x7efec0151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 >> 172.21.15.77:0/17861447 conn(0x7efec006af50 msgr2=0x7efec015e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 shutdown_connections 2023-07-04T18:51:18.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.343+0000 7efec8271700 1 -- 172.21.15.77:0/17861447 wait complete. 2023-07-04T18:51:18.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:51:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.478+0000 7fca212bf700 1 Processor -- start 2023-07-04T18:51:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.478+0000 7fca212bf700 1 -- start start 2023-07-04T18:51:18.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca212bf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0b1860 0x7fca1c0d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca212bf700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fca1c0af8f0 0x7fca1c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca212bf700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1c0b2790 con 0x7fca1c146270 2023-07-04T18:51:18.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca212bf700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1c0b2a70 con 0x7fca1c0af8f0 2023-07-04T18:51:18.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca212bf700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0b2d50 con 0x7fca1c0b1860 2023-07-04T18:51:18.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca1b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fca1c146270 0x7fca1c1602d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50840/0 (socket says 172.21.15.77:50840) 2023-07-04T18:51:18.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca1b7fe700 1 -- 172.21.15.77:0/2492477907 learned_addr learned my addr 172.21.15.77:0/2492477907 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:18.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0b1860 0x7fca1c0d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/2492477907 >> v2:172.21.15.82:3300/0 conn(0x7fca1c0af8f0 0x7fca1c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100807127 0 0) 0x7fca1c0b2790 con 0x7fca1c146270 2023-07-04T18:51:18.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca040035b0 con 0x7fca1c146270 2023-07-04T18:51:18.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3341919763 0 0) 0x7fca040035b0 con 0x7fca1c146270 2023-07-04T18:51:18.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0b1860 msgr2=0x7fca1c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0b1860 0x7fca1c0d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 >> v2:172.21.15.82:3300/0 conn(0x7fca1c0af8f0 msgr2=0x7fca1c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 --2- 172.21.15.77:0/2492477907 >> v2:172.21.15.82:3300/0 conn(0x7fca1c0af8f0 0x7fca1c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.479+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c0b3030 con 0x7fca1c146270 2023-07-04T18:51:18.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fca140019f0 con 0x7fca1c146270 2023-07-04T18:51:18.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/2492477907 >> v2:172.21.15.82:3300/0 conn(0x7fca1c0af8f0 0x7fca1c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:18.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 >> v1:172.21.15.77:6789/0 conn(0x7fca1c146270 legacy=0x7fca1c1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca04003a00 0x7fca04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04004430 0x7fca040048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca04004f60 0x7fca0400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0b2a70 con 0x7fca04003a00 2023-07-04T18:51:18.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0b2d50 con 0x7fca04004430 2023-07-04T18:51:18.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0400bf50 con 0x7fca04004f60 2023-07-04T18:51:18.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0b1860 0x7fca1c0d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:18.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca04003a00 0x7fca04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04004430 0x7fca040048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1b7fe700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca04004f60 0x7fca0400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 -- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca04004f60 msgr2=0x7fca0400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca04004f60 0x7fca0400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 -- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04004430 msgr2=0x7fca040048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04004430 0x7fca040048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 -- 172.21.15.77:0/2492477907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040035b0 con 0x7fca04003a00 2023-07-04T18:51:18.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca1affd700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca04003a00 0x7fca04003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0b5950 tx=0x7fca04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d502cc588163c611 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10004030 con 0x7fca04003a00 2023-07-04T18:51:18.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca10004190 con 0x7fca04003a00 2023-07-04T18:51:18.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.480+0000 7fca19ffb700 1 -- 172.21.15.77:0/2492477907 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10004540 con 0x7fca04003a00 2023-07-04T18:51:18.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 -- 172.21.15.77:0/2492477907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca04003a00 msgr2=0x7fca04003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 --2- 172.21.15.77:0/2492477907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca04003a00 0x7fca04003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0b5950 tx=0x7fca04005880 comp rx=0 tx=0).stop 2023-07-04T18:51:18.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca1b7fe700 1 -- 172.21.15.77:0/2492477907 reap_dead start 2023-07-04T18:51:18.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 -- 172.21.15.77:0/2492477907 shutdown_connections 2023-07-04T18:51:18.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 -- 172.21.15.77:0/2492477907 >> 172.21.15.77:0/2492477907 conn(0x7fca1c071e90 msgr2=0x7fca1c09c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 -- 172.21.15.77:0/2492477907 shutdown_connections 2023-07-04T18:51:18.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.481+0000 7fca212bf700 1 -- 172.21.15.77:0/2492477907 wait complete. 2023-07-04T18:51:18.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.482+0000 7fca212bf700 1 Processor -- start 2023-07-04T18:51:18.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.482+0000 7fca212bf700 1 -- start start 2023-07-04T18:51:18.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04003a00 0x7fca1c15b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0c8a90 0x7fca1c0c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c1f42d0 con 0x7fca1c1efe30 2023-07-04T18:51:18.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c15ba60 con 0x7fca04003a00 2023-07-04T18:51:18.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca212bf700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0c9470 con 0x7fca1c0c8a90 2023-07-04T18:51:18.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43712/0 (socket says 172.21.15.77:43712) 2023-07-04T18:51:18.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 -- 172.21.15.77:0/3111929581 learned_addr learned my addr 172.21.15.77:0/3111929581 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:18.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1affd700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04003a00 0x7fca1c15b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 -- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0c8a90 msgr2=0x7fca1c0c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0c8a90 0x7fca1c0c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 -- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04003a00 msgr2=0x7fca1c15b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04003a00 0x7fca1c15b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 -- 172.21.15.77:0/3111929581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c1f33b0 con 0x7fca1c1efe30 2023-07-04T18:51:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.483+0000 7fca1a7fc700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca0c011cd0 tx=0x7fca0c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.484+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00fa50 con 0x7fca1c1efe30 2023-07-04T18:51:18.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.484+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c00fbb0 con 0x7fca1c1efe30 2023-07-04T18:51:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.484+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c004030 con 0x7fca1c1efe30 2023-07-04T18:51:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.484+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c1f3180 con 0x7fca1c1efe30 2023-07-04T18:51:18.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.484+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c1f3930 con 0x7fca1c1efe30 2023-07-04T18:51:18.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.485+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1c1f4410 con 0x7fca1c1efe30 2023-07-04T18:51:18.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.485+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fca0c004190 con 0x7fca1c1efe30 2023-07-04T18:51:18.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.485+0000 7fca037fe700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 0x7fca0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.485+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fca0c0895a0 con 0x7fca1c1efe30 2023-07-04T18:51:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.486+0000 7fca1affd700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 0x7fca0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.486+0000 7fca1affd700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 0x7fca0806d190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca1c09e190 tx=0x7fca10008070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.490+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fca1c1f4410 con 0x7fca1c1efe30 2023-07-04T18:51:18.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.717+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1c1f4410 con 0x7fca1c1efe30 2023-07-04T18:51:18.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.718+0000 7fca037fe700 1 -- 172.21.15.77:0/3111929581 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3518 (secure 0 0 0) 0x7fca1c1f4410 con 0x7fca1c1efe30 2023-07-04T18:51:18.719 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:18.720 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-04T18:51:18.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.719+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 msgr2=0x7fca0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.719+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 0x7fca0806d190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fca1c09e190 tx=0x7fca10008070 comp rx=0 tx=0).stop 2023-07-04T18:51:18.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.719+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 msgr2=0x7fca1c1f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.719+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca0c011cd0 tx=0x7fca0c00f520 comp rx=0 tx=0).stop 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.719+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 shutdown_connections 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fca0806ace0 0x7fca0806d190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fca1c0c8a90 0x7fca1c0c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca04003a00 0x7fca1c15b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 --2- 172.21.15.77:0/3111929581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1c1efe30 0x7fca1c1f2c40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 >> 172.21.15.77:0/3111929581 conn(0x7fca1c071e90 msgr2=0x7fca1c0b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 shutdown_connections 2023-07-04T18:51:18.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.720+0000 7fca212bf700 1 -- 172.21.15.77:0/3111929581 wait complete. 2023-07-04T18:51:18.724 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2023-07-04T18:51:18.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:18.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.853+0000 7f2d80f6c700 1 Processor -- start 2023-07-04T18:51:18.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.853+0000 7f2d80f6c700 1 -- start start 2023-07-04T18:51:18.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d80f6c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c158c90 0x7f2d7c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d80f6c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d7c0ce400 0x7f2d7c0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d80f6c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c071ad0 con 0x7f2d7c070f60 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d80f6c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0cb9f0 con 0x7f2d7c0ce400 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d80f6c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0c9fd0 con 0x7f2d7c158c90 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d7c0ce400 0x7f2d7c0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2d7c0ce400 0x7f2d7c0ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43722/0 (socket says 172.21.15.77:43722) 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 -- 172.21.15.77:0/3026855309 learned_addr learned my addr 172.21.15.77:0/3026855309 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:18.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d7a59c700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c158c90 0x7f2d7c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 -- 172.21.15.77:0/3026855309 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c158c90 msgr2=0x7f2d7c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c158c90 0x7f2d7c0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 -- 172.21.15.77:0/3026855309 >> v1:172.21.15.82:6789/0 conn(0x7f2d7c070f60 legacy=0x7f2d7c0713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 -- 172.21.15.77:0/3026855309 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c071d50 con 0x7f2d7c0ce400 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d79d9b700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c0ce870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d700098f0 tx=0x7f2d70009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8457617ef82b513 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d7959a700 1 -- 172.21.15.77:0/3026855309 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70004230 con 0x7f2d7c0ce400 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d7959a700 1 -- 172.21.15.77:0/3026855309 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70004390 con 0x7f2d7c0ce400 2023-07-04T18:51:18.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.854+0000 7f2d7959a700 1 -- 172.21.15.77:0/3026855309 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d700036a0 con 0x7f2d7c0ce400 2023-07-04T18:51:18.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 -- 172.21.15.77:0/3026855309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c0ce400 msgr2=0x7f2d7c0ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c0ce870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d700098f0 tx=0x7f2d70009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:18.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 -- 172.21.15.77:0/3026855309 shutdown_connections 2023-07-04T18:51:18.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c158c90 0x7f2d7c0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/3026855309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c0ce870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 -- 172.21.15.77:0/3026855309 >> 172.21.15.77:0/3026855309 conn(0x7f2d7c0cf220 msgr2=0x7f2d7c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:18.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 -- 172.21.15.77:0/3026855309 shutdown_connections 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.855+0000 7f2d80f6c700 1 -- 172.21.15.77:0/3026855309 wait complete. 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.856+0000 7f2d80f6c700 1 Processor -- start 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.856+0000 7f2d80f6c700 1 -- start start 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 0x7f2d7c1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c156af0 con 0x7f2d7c158c90 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b20f0 con 0x7f2d7c0ce400 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d80f6c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b23d0 con 0x7f2d7c070f60 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d7ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d7ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43730/0 (socket says 172.21.15.77:43730) 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d7ad9d700 1 -- 172.21.15.77:0/4090970743 learned_addr learned my addr 172.21.15.77:0/4090970743 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:18.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d79d9b700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.857+0000 7f2d7a59c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 0x7f2d7c1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 -- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 msgr2=0x7f2d7c1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 0x7f2d7c1534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 -- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 msgr2=0x7f2d7c155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 -- 172.21.15.77:0/4090970743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70009550 con 0x7f2d7c158c90 2023-07-04T18:51:18.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d79d9b700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7a59c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 0x7f2d7c1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d7ad9d700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400b970 tx=0x7f2d7400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7400c9b0 con 0x7f2d7c158c90 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7400cb10 con 0x7f2d7c158c90 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c0b26b0 con 0x7f2d7c158c90 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74014440 con 0x7f2d7c158c90 2023-07-04T18:51:18.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.858+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c1539e0 con 0x7f2d7c158c90 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.859+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c0b2bd0 con 0x7f2d7c158c90 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.859+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2d7401d3b0 con 0x7f2d7c158c90 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.860+0000 7f2d6affd700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 0x7f2d5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.860+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2d74088d90 con 0x7f2d7c158c90 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.860+0000 7f2d7a59c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 0x7f2d5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:18.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.860+0000 7f2d7a59c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 0x7f2d5c06d190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0099f0 tx=0x7f2d6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:18.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:18.864+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2d7c0b2bd0 con 0x7f2d7c158c90 2023-07-04T18:51:19.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.050+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d7c0b2bd0 con 0x7f2d7c158c90 2023-07-04T18:51:19.053 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.051+0000 7f2d6affd700 1 -- 172.21.15.77:0/4090970743 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14600 (secure 0 0 0) 0x7f2d7c0b2bd0 con 0x7f2d7c158c90 2023-07-04T18:51:19.053 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:19.053 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:18.312935+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:19.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 msgr2=0x7f2d5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 0x7f2d5c06d190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c0099f0 tx=0x7f2d6c008040 comp rx=0 tx=0).stop 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 msgr2=0x7f2d7c155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400b970 tx=0x7f2d7400bd30 comp rx=0 tx=0).stop 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 shutdown_connections 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2d5c06ace0 0x7f2d5c06d190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2d7c158c90 0x7f2d7c155f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2d7c070f60 0x7f2d7c1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 --2- 172.21.15.77:0/4090970743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d7c0ce400 0x7f2d7c155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 >> 172.21.15.77:0/4090970743 conn(0x7f2d7c0cf220 msgr2=0x7f2d7c0cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 shutdown_connections 2023-07-04T18:51:19.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.053+0000 7f2d80f6c700 1 -- 172.21.15.77:0/4090970743 wait complete. 2023-07-04T18:51:19.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:19.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:19.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.188+0000 7f94d1e8a700 1 Processor -- start 2023-07-04T18:51:19.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.188+0000 7f94d1e8a700 1 -- start start 2023-07-04T18:51:19.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94d1e8a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f94cc1590b0 0x7f94cc15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94d1e8a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94cc06cfc0 0x7f94cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94d1e8a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc0b3f60 con 0x7f94cc0b3330 2023-07-04T18:51:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94d1e8a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc06caa0 con 0x7f94cc06cfc0 2023-07-04T18:51:19.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94d1e8a700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc0b4b00 con 0x7f94cc1590b0 2023-07-04T18:51:19.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94cbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94cc06cfc0 0x7f94cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94caffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f94cc0b3330 0x7f94cc0b43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50874/0 (socket says 172.21.15.77:50874) 2023-07-04T18:51:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94caffd700 1 -- 172.21.15.77:0/3231621532 learned_addr learned my addr 172.21.15.77:0/3231621532 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.189+0000 7f94cb7fe700 1 --2- 172.21.15.77:0/3231621532 >> v2:172.21.15.82:3300/0 conn(0x7f94cc1590b0 0x7f94cc15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94cbfff700 1 -- 172.21.15.77:0/3231621532 >> v2:172.21.15.82:3300/0 conn(0x7f94cc1590b0 msgr2=0x7f94cc15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:19.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94cbfff700 1 --2- 172.21.15.77:0/3231621532 >> v2:172.21.15.82:3300/0 conn(0x7f94cc1590b0 0x7f94cc15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94cbfff700 1 -- 172.21.15.77:0/3231621532 >> v1:172.21.15.77:6789/0 conn(0x7f94cc0b3330 legacy=0x7f94cc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94ca7fc700 1 -- 172.21.15.77:0/3231621532 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721644580 0 0) 0x7f94cc0b3f60 con 0x7f94cc0b3330 2023-07-04T18:51:19.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94cbfff700 1 -- 172.21.15.77:0/3231621532 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc15dc10 con 0x7f94cc06cfc0 2023-07-04T18:51:19.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94cbfff700 1 --2- 172.21.15.77:0/3231621532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc06cfc0 0x7f94cc06d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0b26d0 tx=0x7f94c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb34afd95377dcb server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94ca7fc700 1 -- 172.21.15.77:0/3231621532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c400e070 con 0x7f94cc06cfc0 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94ca7fc700 1 -- 172.21.15.77:0/3231621532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c400bbd0 con 0x7f94cc06cfc0 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94ca7fc700 1 -- 172.21.15.77:0/3231621532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4012430 con 0x7f94cc06cfc0 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 -- 172.21.15.77:0/3231621532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc06cfc0 msgr2=0x7f94cc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/3231621532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc06cfc0 0x7f94cc06d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0b26d0 tx=0x7f94c400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 -- 172.21.15.77:0/3231621532 shutdown_connections 2023-07-04T18:51:19.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/3231621532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc06cfc0 0x7f94cc06d3c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/3231621532 >> v2:172.21.15.82:3300/0 conn(0x7f94cc1590b0 0x7f94cc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 -- 172.21.15.77:0/3231621532 >> 172.21.15.77:0/3231621532 conn(0x7f94cc06af50 msgr2=0x7f94cc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 -- 172.21.15.77:0/3231621532 shutdown_connections 2023-07-04T18:51:19.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.190+0000 7f94d1e8a700 1 -- 172.21.15.77:0/3231621532 wait complete. 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.192+0000 7f94d1e8a700 1 Processor -- start 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.192+0000 7f94d1e8a700 1 -- start start 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94cc06cfc0 0x7f94cc0ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94cc0b3330 0x7f94cc15aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc1f0140 con 0x7f94cc1590b0 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc1f0420 con 0x7f94cc06cfc0 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94d1e8a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc1f0700 con 0x7f94cc0b3330 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43744/0 (socket says 172.21.15.77:43744) 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 -- 172.21.15.77:0/1310028646 learned_addr learned my addr 172.21.15.77:0/1310028646 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cb7fe700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94cc06cfc0 0x7f94cc0ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94caffd700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94cc0b3330 0x7f94cc15aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 -- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94cc0b3330 msgr2=0x7f94cc15aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94cc0b3330 0x7f94cc15aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 -- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94cc06cfc0 msgr2=0x7f94cc0ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.193+0000 7f94cbfff700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94cc06cfc0 0x7f94cc0ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94cbfff700 1 -- 172.21.15.77:0/1310028646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4009a70 con 0x7f94cc1590b0 2023-07-04T18:51:19.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94cbfff700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94cc071970 tx=0x7f94c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4012430 con 0x7f94cc1590b0 2023-07-04T18:51:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4020070 con 0x7f94cc1590b0 2023-07-04T18:51:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4004280 con 0x7f94cc1590b0 2023-07-04T18:51:19.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc1f09e0 con 0x7f94cc1590b0 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.194+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc15b430 con 0x7f94cc1590b0 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.195+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc06d8a0 con 0x7f94cc1590b0 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.195+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94c40043e0 con 0x7f94cc1590b0 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.195+0000 7f94b3fff700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 0x7f94b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.195+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f94c4022070 con 0x7f94cc1590b0 2023-07-04T18:51:19.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.196+0000 7f94cb7fe700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 0x7f94b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.196+0000 7f94cb7fe700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 0x7f94b806d190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94cc15bd40 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.200+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f94c4056200 con 0x7f94cc1590b0 2023-07-04T18:51:19.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.390+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94cc04e210 con 0x7f94cc1590b0 2023-07-04T18:51:19.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.390+0000 7f94b3fff700 1 -- 172.21.15.77:0/1310028646 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94c402c0a0 con 0x7f94cc1590b0 2023-07-04T18:51:19.392 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:19.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 msgr2=0x7f94b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 0x7f94b806d190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94cc15bd40 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2023-07-04T18:51:19.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 msgr2=0x7f94cc0aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94cc071970 tx=0x7f94c4004ad0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 shutdown_connections 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94b806ace0 0x7f94b806d190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94cc1590b0 0x7f94cc0aa4b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94cc06cfc0 0x7f94cc0ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 --2- 172.21.15.77:0/1310028646 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94cc0b3330 0x7f94cc15aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 >> 172.21.15.77:0/1310028646 conn(0x7f94cc06af50 msgr2=0x7f94cc153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 shutdown_connections 2023-07-04T18:51:19.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.392+0000 7f94d1e8a700 1 -- 172.21.15.77:0/1310028646 wait complete. 2023-07-04T18:51:19.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:19.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.527+0000 7ff5e178a700 1 Processor -- start 2023-07-04T18:51:19.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.527+0000 7ff5e178a700 1 -- start start 2023-07-04T18:51:19.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5e178a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5dc06ec20 0x7ff5dc15d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5e178a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc160a60 0x7ff5dc069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5e178a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc15da60 con 0x7ff5dc06a380 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5e178a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc06a800 con 0x7ff5dc06ec20 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5e178a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc06eff0 con 0x7ff5dc160a60 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5daffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5dc06ec20 0x7ff5dc15d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5db7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff5dc06a380 0x7ff5dc0b2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50890/0 (socket says 172.21.15.77:50890) 2023-07-04T18:51:19.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5db7fe700 1 -- 172.21.15.77:0/2503819814 learned_addr learned my addr 172.21.15.77:0/2503819814 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5da7fc700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc160a60 0x7ff5dc069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5daffd700 1 -- 172.21.15.77:0/2503819814 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc160a60 msgr2=0x7ff5dc069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5daffd700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc160a60 0x7ff5dc069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5daffd700 1 -- 172.21.15.77:0/2503819814 >> v1:172.21.15.77:6789/0 conn(0x7ff5dc06a380 legacy=0x7ff5dc0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5daffd700 1 -- 172.21.15.77:0/2503819814 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0ac8a0 con 0x7ff5dc06ec20 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.528+0000 7ff5d9ffb700 1 -- 172.21.15.77:0/2503819814 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071317624 0 0) 0x7ff5d4001490 con 0x7ff5dc06a380 2023-07-04T18:51:19.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5daffd700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc06ec20 0x7ff5dc15d520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e5a5c90239166c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5d9ffb700 1 -- 172.21.15.77:0/2503819814 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011830 con 0x7ff5dc06ec20 2023-07-04T18:51:19.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5d9ffb700 1 -- 172.21.15.77:0/2503819814 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011990 con 0x7ff5dc06ec20 2023-07-04T18:51:19.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5d9ffb700 1 -- 172.21.15.77:0/2503819814 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011d20 con 0x7ff5dc06ec20 2023-07-04T18:51:19.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5e178a700 1 -- 172.21.15.77:0/2503819814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc06ec20 msgr2=0x7ff5dc15d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.529+0000 7ff5e178a700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc06ec20 0x7ff5dc15d520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009840 tx=0x7ff5cc009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:19.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 -- 172.21.15.77:0/2503819814 shutdown_connections 2023-07-04T18:51:19.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc06ec20 0x7ff5dc15d520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 --2- 172.21.15.77:0/2503819814 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc160a60 0x7ff5dc069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 -- 172.21.15.77:0/2503819814 >> 172.21.15.77:0/2503819814 conn(0x7ff5dc06af50 msgr2=0x7ff5dc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 -- 172.21.15.77:0/2503819814 shutdown_connections 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.530+0000 7ff5e178a700 1 -- 172.21.15.77:0/2503819814 wait complete. 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.531+0000 7ff5e178a700 1 Processor -- start 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.531+0000 7ff5e178a700 1 -- start start 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5dc06a380 0x7ff5dc1e8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc06ec20 0x7ff5dc151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cf2a0 con 0x7ff5dc160a60 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cf4f0 con 0x7ff5dc06a380 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0cf7d0 con 0x7ff5dc06ec20 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43758/0 (socket says 172.21.15.77:43758) 2023-07-04T18:51:19.538 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 -- 172.21.15.77:0/19246180 learned_addr learned my addr 172.21.15.77:0/19246180 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:19.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5daffd700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5dc06a380 0x7ff5dc1e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5da7fc700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc06ec20 0x7ff5dc151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 -- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc06ec20 msgr2=0x7ff5dc151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc06ec20 0x7ff5dc151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 -- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5dc06a380 msgr2=0x7ff5dc1e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5dc06a380 0x7ff5dc1e8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 -- 172.21.15.77:0/19246180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc0094a0 con 0x7ff5dc160a60 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5db7fe700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400cca0 tx=0x7ff5d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400ab10 con 0x7ff5dc160a60 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d400ac90 con 0x7ff5dc160a60 2023-07-04T18:51:19.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0cfab0 con 0x7ff5dc160a60 2023-07-04T18:51:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40070d0 con 0x7ff5dc160a60 2023-07-04T18:51:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.532+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0d0070 con 0x7ff5dc160a60 2023-07-04T18:51:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.533+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc0add60 con 0x7ff5dc160a60 2023-07-04T18:51:19.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.534+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff5d4007250 con 0x7ff5dc160a60 2023-07-04T18:51:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.534+0000 7ff5c37fe700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 0x7ff5c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.534+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff5d4088cc0 con 0x7ff5dc160a60 2023-07-04T18:51:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.534+0000 7ff5daffd700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 0x7ff5c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.535+0000 7ff5daffd700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 0x7ff5c806d190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0090b0 tx=0x7ff5cc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.538+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff5d4054150 con 0x7ff5dc160a60 2023-07-04T18:51:19.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.728+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff5dc15da60 con 0x7ff5dc160a60 2023-07-04T18:51:19.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.728+0000 7ff5c37fe700 1 -- 172.21.15.77:0/19246180 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff5dc15da60 con 0x7ff5dc160a60 2023-07-04T18:51:19.734 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:19.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 msgr2=0x7ff5c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 0x7ff5c806d190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0090b0 tx=0x7ff5cc008030 comp rx=0 tx=0).stop 2023-07-04T18:51:19.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 msgr2=0x7ff5dc0b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400cca0 tx=0x7ff5d400a060 comp rx=0 tx=0).stop 2023-07-04T18:51:19.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 shutdown_connections 2023-07-04T18:51:19.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff5c806ace0 0x7ff5c806d190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5dc160a60 0x7ff5dc0b1180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5dc06a380 0x7ff5dc1e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 --2- 172.21.15.77:0/19246180 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5dc06ec20 0x7ff5dc151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 >> 172.21.15.77:0/19246180 conn(0x7ff5dc06af50 msgr2=0x7ff5dc0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.730+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 shutdown_connections 2023-07-04T18:51:19.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.731+0000 7ff5e178a700 1 -- 172.21.15.77:0/19246180 wait complete. 2023-07-04T18:51:19.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-04T18:51:19.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.862+0000 7f8d8d0cb700 1 Processor -- start 2023-07-04T18:51:19.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.862+0000 7f8d8d0cb700 1 -- start start 2023-07-04T18:51:19.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8d0cb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8d0cb700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f8d880cf920 0x7f8d8806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8d0cb700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d880cf100 con 0x7f8d8815a430 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8d0cb700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8815df60 con 0x7f8d880cf920 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8d0cb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8815e240 con 0x7f8d8815ea40 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8659c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8759e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f8d880cf920 0x7f8d8806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8659c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33616/0 (socket says 172.21.15.77:33616) 2023-07-04T18:51:19.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.863+0000 7f8d8659c700 1 -- 172.21.15.77:0/301395759 learned_addr learned my addr 172.21.15.77:0/301395759 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d8659c700 1 -- 172.21.15.77:0/301395759 >> v2:172.21.15.82:3300/0 conn(0x7f8d880cf920 msgr2=0x7f8d8806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d8659c700 1 --2- 172.21.15.77:0/301395759 >> v2:172.21.15.82:3300/0 conn(0x7f8d880cf920 0x7f8d8806ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d8659c700 1 -- 172.21.15.77:0/301395759 >> v1:172.21.15.82:6789/0 conn(0x7f8d8815a430 legacy=0x7f8d880ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d8659c700 1 -- 172.21.15.77:0/301395759 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8815e520 con 0x7f8d8815ea40 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d8659c700 1 --2- 172.21.15.77:0/301395759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8d780098f0 tx=0x7f8d78009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc9cc74f5f754c7 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d85d9b700 1 -- 172.21.15.77:0/301395759 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78011830 con 0x7f8d8815ea40 2023-07-04T18:51:19.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d85d9b700 1 -- 172.21.15.77:0/301395759 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78011990 con 0x7f8d8815ea40 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.864+0000 7f8d85d9b700 1 -- 172.21.15.77:0/301395759 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78011d20 con 0x7f8d8815ea40 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/301395759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 msgr2=0x7f8d880cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/301395759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8d780098f0 tx=0x7f8d78009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/301395759 shutdown_connections 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/301395759 >> v2:172.21.15.82:3300/0 conn(0x7f8d880cf920 0x7f8d8806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/301395759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8815ea40 0x7f8d880cf3e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/301395759 >> 172.21.15.77:0/301395759 conn(0x7f8d8806d7c0 msgr2=0x7f8d8806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:19.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/301395759 shutdown_connections 2023-07-04T18:51:19.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.865+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/301395759 wait complete. 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.866+0000 7f8d8d0cb700 1 Processor -- start 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.866+0000 7f8d8d0cb700 1 -- start start 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d880cf920 0x7f8d88152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8806ccf0 0x7f8d8815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d881e4e30 con 0x7f8d8815ea40 2023-07-04T18:51:19.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d881e50d0 con 0x7f8d880cf920 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8d0cb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d881e53b0 con 0x7f8d8806ccf0 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43770/0 (socket says 172.21.15.77:43770) 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 -- 172.21.15.77:0/115490934 learned_addr learned my addr 172.21.15.77:0/115490934 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d86d9d700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d880cf920 0x7f8d88152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8759e700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8806ccf0 0x7f8d8815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 -- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8806ccf0 msgr2=0x7f8d8815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8806ccf0 0x7f8d8815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 -- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d880cf920 msgr2=0x7f8d88152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d880cf920 0x7f8d88152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.867+0000 7f8d8659c700 1 -- 172.21.15.77:0/115490934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009550 con 0x7f8d8815ea40 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d8659c700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009c40 tx=0x7f8d78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78004550 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7801f070 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d881e5690 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780038e0 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.868+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d881e5c50 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.869+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8804e210 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.869+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d78003a40 con 0x7f8d8815ea40 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.870+0000 7f8d6f7fe700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 0x7f8d7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:19.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.870+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d78021070 con 0x7f8d8815ea40 2023-07-04T18:51:19.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.870+0000 7f8d86d9d700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 0x7f8d7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:19.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.870+0000 7f8d86d9d700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 0x7f8d7406d190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c007fd0 tx=0x7f8d7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:19.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:19.874+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d78055dd0 con 0x7f8d8815ea40 2023-07-04T18:51:20.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.060+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8d880ce270 con 0x7f8d8815ea40 2023-07-04T18:51:20.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.061+0000 7f8d6f7fe700 1 -- 172.21.15.77:0/115490934 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7f8d78015090 con 0x7f8d8815ea40 2023-07-04T18:51:20.062 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 2, 2023-07-04T18:51:20.063 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": "rbd" 2023-07-04T18:51:20.064 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:51:20.064 INFO:teuthology.orchestra.run.smithi077.stdout:] 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 msgr2=0x7f8d7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 0x7f8d7406d190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c007fd0 tx=0x7f8d7c009040 comp rx=0 tx=0).stop 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 msgr2=0x7f8d880ad1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009c40 tx=0x7f8d78004750 comp rx=0 tx=0).stop 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 shutdown_connections 2023-07-04T18:51:20.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8d7406ace0 0x7f8d7406d190 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8d8806ccf0 0x7f8d8815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d880cf920 0x7f8d88152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 --2- 172.21.15.77:0/115490934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d8815ea40 0x7f8d880ad1c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 >> 172.21.15.77:0/115490934 conn(0x7f8d8806d7c0 msgr2=0x7f8d88157a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:20.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 shutdown_connections 2023-07-04T18:51:20.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.063+0000 7f8d8d0cb700 1 -- 172.21.15.77:0/115490934 wait complete. 2023-07-04T18:51:20.076 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-04T18:51:20.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-04T18:51:20.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.199+0000 7f078c497700 1 Processor -- start 2023-07-04T18:51:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.199+0000 7f078c497700 1 -- start start 2023-07-04T18:51:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078c497700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 0x7f07840b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078c497700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f078406a550 0x7f0784160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078c497700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078415ca10 con 0x7f07840aec80 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078c497700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07840cdc90 con 0x7f078406a550 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078c497700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0784160210 con 0x7f078415c640 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078aa34700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f078406a550 0x7f0784160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f0789a32700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f07840aec80 0x7f0784069e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50914/0 (socket says 172.21.15.77:50914) 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f0789a32700 1 -- 172.21.15.77:0/198636601 learned_addr learned my addr 172.21.15.77:0/198636601 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078a233700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 0x7f07840b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078aa34700 1 -- 172.21.15.77:0/198636601 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 msgr2=0x7f07840b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078aa34700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 0x7f07840b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078aa34700 1 -- 172.21.15.77:0/198636601 >> v1:172.21.15.77:6789/0 conn(0x7f07840aec80 legacy=0x7f0784069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078aa34700 1 -- 172.21.15.77:0/198636601 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07841604f0 con 0x7f078406a550 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f0789231700 1 -- 172.21.15.77:0/198636601 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769663289 0 0) 0x7f078415ca10 con 0x7f07840aec80 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.200+0000 7f078a233700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 0x7f07840b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078aa34700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f0784160cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f077c009d90 tx=0x7f077c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b037a467693b8d4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f0789231700 1 -- 172.21.15.77:0/198636601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c00e070 con 0x7f078406a550 2023-07-04T18:51:20.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f0789231700 1 -- 172.21.15.77:0/198636601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c00bbd0 con 0x7f078406a550 2023-07-04T18:51:20.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f0789231700 1 -- 172.21.15.77:0/198636601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c012430 con 0x7f078406a550 2023-07-04T18:51:20.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 -- 172.21.15.77:0/198636601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 msgr2=0x7f0784160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f0784160cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f077c009d90 tx=0x7f077c00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:20.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 -- 172.21.15.77:0/198636601 shutdown_connections 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f0784160cf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 --2- 172.21.15.77:0/198636601 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f078415c640 0x7f07840b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 -- 172.21.15.77:0/198636601 >> 172.21.15.77:0/198636601 conn(0x7f078406bbe0 msgr2=0x7f07840b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 -- 172.21.15.77:0/198636601 shutdown_connections 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.201+0000 7f078c497700 1 -- 172.21.15.77:0/198636601 wait complete. 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 Processor -- start 2023-07-04T18:51:20.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 -- start start 2023-07-04T18:51:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07840aec80 0x7f07840b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 0x7f07840b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07841e6070 con 0x7f078406a550 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07841e6350 con 0x7f078415c640 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078c497700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07841e6630 con 0x7f07840aec80 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078a233700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078a233700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43780/0 (socket says 172.21.15.77:43780) 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.203+0000 7f078a233700 1 -- 172.21.15.77:0/2405835923 learned_addr learned my addr 172.21.15.77:0/2405835923 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078aa34700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 0x7f07840b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f0789a32700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07840aec80 0x7f07840b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 -- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07840aec80 msgr2=0x7f07840b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07840aec80 0x7f07840b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 -- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 msgr2=0x7f07840b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 0x7f07840b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 -- 172.21.15.77:0/2405835923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c009a70 con 0x7f078406a550 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078aa34700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 0x7f07840b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078a233700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f077400e810 tx=0x7f077400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077400c020 con 0x7f078406a550 2023-07-04T18:51:20.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774004250 con 0x7f078406a550 2023-07-04T18:51:20.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07841e6910 con 0x7f078406a550 2023-07-04T18:51:20.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07740036a0 con 0x7f078406a550 2023-07-04T18:51:20.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.204+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0784150320 con 0x7f078406a550 2023-07-04T18:51:20.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.205+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07841e6b00 con 0x7f078406a550 2023-07-04T18:51:20.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.205+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0774003800 con 0x7f078406a550 2023-07-04T18:51:20.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.206+0000 7f077affd700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 0x7f076806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.206+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0774088ea0 con 0x7f078406a550 2023-07-04T18:51:20.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.206+0000 7f0789a32700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 0x7f076806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.206+0000 7f0789a32700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 0x7f076806d1c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0784161060 tx=0x7f0780007be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.210+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f07740542a0 con 0x7f078406a550 2023-07-04T18:51:20.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.400+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f07841e61b0 con 0x7f078406a550 2023-07-04T18:51:20.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.446+0000 7f077affd700 1 -- 172.21.15.77:0/2405835923 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f07841e61b0 con 0x7f078406a550 2023-07-04T18:51:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 msgr2=0x7f076806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 0x7f076806d1c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0784161060 tx=0x7f0780007be0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 msgr2=0x7f078414fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f077400e810 tx=0x7f077400ebd0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 shutdown_connections 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f076806ad10 0x7f076806d1c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f078406a550 0x7f078414fde0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f078415c640 0x7f07840b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 --2- 172.21.15.77:0/2405835923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07840aec80 0x7f07840b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 >> 172.21.15.77:0/2405835923 conn(0x7f078406bbe0 msgr2=0x7f07840b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:20.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 shutdown_connections 2023-07-04T18:51:20.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.450+0000 7f078c497700 1 -- 172.21.15.77:0/2405835923 wait complete. 2023-07-04T18:51:20.455 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2023-07-04T18:51:20.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-04T18:51:20.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.590+0000 7f0b755ac700 1 Processor -- start 2023-07-04T18:51:20.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.590+0000 7f0b755ac700 1 -- start start 2023-07-04T18:51:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b755ac700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0b700aeac0 0x7f0b700cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b755ac700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7015a820 con 0x7f0b7015a450 2023-07-04T18:51:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b755ac700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7015eb70 con 0x7f0b7006a550 2023-07-04T18:51:20.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b755ac700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7015df60 con 0x7f0b700aeac0 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b6effd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0b7015a450 0x7f0b70069e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50926/0 (socket says 172.21.15.77:50926) 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b6effd700 1 -- 172.21.15.77:0/1205488764 learned_addr learned my addr 172.21.15.77:0/1205488764 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.591+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/1205488764 >> v2:172.21.15.82:3300/0 conn(0x7f0b700aeac0 0x7f0b700cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291651113 0 0) 0x7f0b64001110 con 0x7f0b7015a450 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580035b0 con 0x7f0b7015a450 2023-07-04T18:51:20.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918696279 0 0) 0x7f0b580035b0 con 0x7f0b7015a450 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 >> v2:172.21.15.82:3300/0 conn(0x7f0b700aeac0 msgr2=0x7f0b700cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 --2- 172.21.15.77:0/1205488764 >> v2:172.21.15.82:3300/0 conn(0x7f0b700aeac0 0x7f0b700cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 >> v1:172.21.15.82:6789/0 conn(0x7f0b7006a550 legacy=0x7f0b700cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7015e240 con 0x7f0b7015a450 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f0b64001800 con 0x7f0b7015a450 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 >> v1:172.21.15.77:6789/0 conn(0x7f0b7015a450 legacy=0x7f0b70069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b58003a00 0x7f0b58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b580043f0 0x7f0b580048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 0x7f0b5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015df60 con 0x7f0b580050b0 2023-07-04T18:51:20.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c000 con 0x7f0b58003a00 2023-07-04T18:51:20.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5800c1c0 con 0x7f0b580043f0 2023-07-04T18:51:20.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 0x7f0b5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6effd700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b58003a00 0x7f0b58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6e7fc700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b580043f0 0x7f0b580048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b580043f0 msgr2=0x7f0b580048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b580043f0 0x7f0b580048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b58003a00 msgr2=0x7f0b58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:20.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b58003a00 0x7f0b58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/1205488764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b580035b0 con 0x7f0b580050b0 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/1205488764 reap_dead start 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.592+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 0x7f0b5800ba10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fc0 tx=0x7f0b58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2b34030a1cfea0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68003a20 con 0x7f0b580050b0 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6800f040 con 0x7f0b580050b0 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b6dffb700 1 -- 172.21.15.77:0/1205488764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68003de0 con 0x7f0b580050b0 2023-07-04T18:51:20.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 -- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 msgr2=0x7f0b5800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 0x7f0b5800ba10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009fc0 tx=0x7f0b58005880 comp rx=0 tx=0).stop 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 -- 172.21.15.77:0/1205488764 shutdown_connections 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 --2- 172.21.15.77:0/1205488764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b580050b0 0x7f0b5800ba10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 -- 172.21.15.77:0/1205488764 >> 172.21.15.77:0/1205488764 conn(0x7f0b7006bba0 msgr2=0x7f0b700b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 -- 172.21.15.77:0/1205488764 shutdown_connections 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.593+0000 7f0b755ac700 1 -- 172.21.15.77:0/1205488764 wait complete. 2023-07-04T18:51:20.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.594+0000 7f0b755ac700 1 Processor -- start 2023-07-04T18:51:20.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 -- start start 2023-07-04T18:51:20.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b580050b0 0x7f0b700caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 0x7f0b700c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700aca50 con 0x7f0b7015b0c0 2023-07-04T18:51:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700ce900 con 0x7f0b580050b0 2023-07-04T18:51:20.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b755ac700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700cdd80 con 0x7f0b700c8c60 2023-07-04T18:51:20.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43804/0 (socket says 172.21.15.77:43804) 2023-07-04T18:51:20.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/887988208 learned_addr learned my addr 172.21.15.77:0/887988208 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:20.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b6effd700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b580050b0 0x7f0b700caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.595+0000 7f0b6e7fc700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 0x7f0b700c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 msgr2=0x7f0b700c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 0x7f0b700c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b580050b0 msgr2=0x7f0b700caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:20.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b580050b0 0x7f0b700caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:20.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 -- 172.21.15.77:0/887988208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700ce060 con 0x7f0b7015b0c0 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6e7fc700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 0x7f0b700c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b6f7fe700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b700d0a90 tx=0x7f0b68009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68003e10 con 0x7f0b7015b0c0 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b68004230 con 0x7f0b7015b0c0 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680045e0 con 0x7f0b7015b0c0 2023-07-04T18:51:20.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b700ce340 con 0x7f0b7015b0c0 2023-07-04T18:51:20.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.596+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b700cb4b0 con 0x7f0b7015b0c0 2023-07-04T18:51:20.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.597+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b700acb90 con 0x7f0b7015b0c0 2023-07-04T18:51:20.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.597+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0b68024070 con 0x7f0b7015b0c0 2023-07-04T18:51:20.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.598+0000 7f0b577fe700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 0x7f0b5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:20.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.598+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f0b6808a750 con 0x7f0b7015b0c0 2023-07-04T18:51:20.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.598+0000 7f0b6effd700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 0x7f0b5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:20.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.598+0000 7f0b6effd700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 0x7f0b5c06d190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b64004b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:20.618 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.602+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0b700acb90 con 0x7f0b7015b0c0 2023-07-04T18:51:20.788 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:20.787+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0b700acb90 con 0x7f0b7015b0c0 2023-07-04T18:51:21.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:51:21.178 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:51:21.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.448+0000 7f0b577fe700 1 -- 172.21.15.77:0/887988208 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0b700acb90 con 0x7f0b7015b0c0 2023-07-04T18:51:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 msgr2=0x7f0b5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 0x7f0b5c06d190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b64004b50 comp rx=0 tx=0).stop 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 msgr2=0x7f0b7015b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0b700d0a90 tx=0x7f0b68009750 comp rx=0 tx=0).stop 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 shutdown_connections 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0b5c06ace0 0x7f0b5c06d190 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b7015b0c0 0x7f0b7015b530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b580050b0 0x7f0b700caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 --2- 172.21.15.77:0/887988208 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0b700c8c60 0x7f0b700c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 >> 172.21.15.77:0/887988208 conn(0x7f0b7006bba0 msgr2=0x7f0b700b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.452+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 shutdown_connections 2023-07-04T18:51:21.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.453+0000 7f0b755ac700 1 -- 172.21.15.77:0/887988208 wait complete. 2023-07-04T18:51:21.465 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2023-07-04T18:51:21.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-04T18:51:21.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.600+0000 7f9193df1700 1 Processor -- start 2023-07-04T18:51:21.602 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.600+0000 7f9193df1700 1 -- start start 2023-07-04T18:51:21.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9193df1700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9193df1700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f918c06cfc0 0x7f918c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9193df1700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c0b3f60 con 0x7f918c1590b0 2023-07-04T18:51:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9193df1700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c06caa0 con 0x7f918c06cfc0 2023-07-04T18:51:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9193df1700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c06d940 con 0x7f918c0b3330 2023-07-04T18:51:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919138c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919238e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f918c06cfc0 0x7f918c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919138c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33660/0 (socket says 172.21.15.77:33660) 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f9191b8d700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f918c1590b0 0x7f918c15d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:58858/0 (socket says 172.21.15.77:58858) 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919138c700 1 -- 172.21.15.77:0/3108915875 learned_addr learned my addr 172.21.15.77:0/3108915875 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919238e700 1 -- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 msgr2=0x7f918c0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919238e700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919238e700 1 -- 172.21.15.77:0/3108915875 >> v1:172.21.15.82:6789/0 conn(0x7f918c1590b0 legacy=0x7f918c15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919238e700 1 -- 172.21.15.77:0/3108915875 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c06c4e0 con 0x7f918c06cfc0 2023-07-04T18:51:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.601+0000 7f919138c700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f919238e700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c06d400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9184009d90 tx=0x7f918400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85ddf13b44d369f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9190b8b700 1 -- 172.21.15.77:0/3108915875 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400e070 con 0x7f918c06cfc0 2023-07-04T18:51:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9190b8b700 1 -- 172.21.15.77:0/3108915875 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918400bb20 con 0x7f918c06cfc0 2023-07-04T18:51:21.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9190b8b700 1 -- 172.21.15.77:0/3108915875 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184012430 con 0x7f918c06cfc0 2023-07-04T18:51:21.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 -- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 msgr2=0x7f918c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c06d400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9184009d90 tx=0x7f918400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 -- 172.21.15.77:0/3108915875 shutdown_connections 2023-07-04T18:51:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c06d400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 --2- 172.21.15.77:0/3108915875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c0b3330 0x7f918c0b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 -- 172.21.15.77:0/3108915875 >> 172.21.15.77:0/3108915875 conn(0x7f918c06af50 msgr2=0x7f918c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 -- 172.21.15.77:0/3108915875 shutdown_connections 2023-07-04T18:51:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.602+0000 7f9193df1700 1 -- 172.21.15.77:0/3108915875 wait complete. 2023-07-04T18:51:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.603+0000 7f9193df1700 1 Processor -- start 2023-07-04T18:51:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 -- start start 2023-07-04T18:51:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c1505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c1590b0 0x7f918c1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0b1b10 con 0x7f918c0b3330 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0b02d0 con 0x7f918c06cfc0 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f9193df1700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0afcc0 con 0x7f918c1590b0 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f919138c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f919138c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43816/0 (socket says 172.21.15.77:43816) 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.604+0000 7f919138c700 1 -- 172.21.15.77:0/825079023 learned_addr learned my addr 172.21.15.77:0/825079023 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:21.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f9191b8d700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c1505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919238e700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c1590b0 0x7f918c1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 -- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c1590b0 msgr2=0x7f918c1436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c1590b0 0x7f918c1436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 -- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 msgr2=0x7f918c1505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c1505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 -- 172.21.15.77:0/825079023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c009550 con 0x7f918c0b3330 2023-07-04T18:51:21.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f919138c700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f917c00ebe0 tx=0x7f917c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00c850 con 0x7f918c0b3330 2023-07-04T18:51:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c00c9b0 con 0x7f918c0b3330 2023-07-04T18:51:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9184009a70 con 0x7f918c0b3330 2023-07-04T18:51:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00cd40 con 0x7f918c0b3330 2023-07-04T18:51:21.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.605+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c0aff60 con 0x7f918c0b3330 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.606+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c0b0440 con 0x7f918c0b3330 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.607+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f917c013470 con 0x7f918c0b3330 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.607+0000 7f91827fc700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 0x7f917006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.607+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f917c0894d0 con 0x7f918c0b3330 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.607+0000 7f9191b8d700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 0x7f917006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:21.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.608+0000 7f9191b8d700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 0x7f917006d190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9188000f60 tx=0x7f9188007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:21.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.611+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f917c054670 con 0x7f918c0b3330 2023-07-04T18:51:21.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:21.843+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f918c04e210 con 0x7f918c0b3330 2023-07-04T18:51:22.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.477+0000 7f91827fc700 1 -- 172.21.15.77:0/825079023 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f917c057c90 con 0x7f918c0b3330 2023-07-04T18:51:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 msgr2=0x7f917006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 0x7f917006d190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9188000f60 tx=0x7f9188007ed0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 msgr2=0x7f918c0b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f917c00ebe0 tx=0x7f917c00eef0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 shutdown_connections 2023-07-04T18:51:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f917006ace0 0x7f917006d190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f918c1590b0 0x7f918c1436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f918c06cfc0 0x7f918c1505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 --2- 172.21.15.77:0/825079023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f918c0b3330 0x7f918c0b0d70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 >> 172.21.15.77:0/825079023 conn(0x7f918c06af50 msgr2=0x7f918c15bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:22.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 shutdown_connections 2023-07-04T18:51:22.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.480+0000 7f9193df1700 1 -- 172.21.15.77:0/825079023 wait complete. 2023-07-04T18:51:22.484 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 3 and data pool 4 2023-07-04T18:51:22.501 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:22.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.620+0000 7ff40cee8700 1 Processor -- start 2023-07-04T18:51:22.622 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.620+0000 7ff40cee8700 1 -- start start 2023-07-04T18:51:22.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40cee8700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff4080ce4a0 0x7ff40815b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40cee8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff40815e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40cee8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40815b080 con 0x7ff4080cbd00 2023-07-04T18:51:22.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40cee8700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4080ce870 con 0x7ff4080ce4a0 2023-07-04T18:51:22.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40cee8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40815ea30 con 0x7ff40815dfb0 2023-07-04T18:51:22.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff406d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff4080cbd00 0x7ff40806a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50964/0 (socket says 172.21.15.77:50964) 2023-07-04T18:51:22.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff406d9d700 1 -- 172.21.15.77:0/1245929784 learned_addr learned my addr 172.21.15.77:0/1245929784 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:22.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40659c700 1 --2- 172.21.15.77:0/1245929784 >> v2:172.21.15.82:3300/0 conn(0x7ff4080ce4a0 0x7ff40815b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff405d9b700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff40815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988497873 0 0) 0x7ff40815b080 con 0x7ff4080cbd00 2023-07-04T18:51:22.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f00035b0 con 0x7ff4080cbd00 2023-07-04T18:51:22.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273518437 0 0) 0x7ff3f00035b0 con 0x7ff4080cbd00 2023-07-04T18:51:22.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 msgr2=0x7ff40815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:22.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff40815e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 >> v2:172.21.15.82:3300/0 conn(0x7ff4080ce4a0 msgr2=0x7ff40815b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:22.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 --2- 172.21.15.77:0/1245929784 >> v2:172.21.15.82:3300/0 conn(0x7ff4080ce4a0 0x7ff40815b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40815ed10 con 0x7ff4080cbd00 2023-07-04T18:51:22.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7ff4000019f0 con 0x7ff4080cbd00 2023-07-04T18:51:22.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 >> v1:172.21.15.77:6789/0 conn(0x7ff4080cbd00 legacy=0x7ff40806a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f0003a00 0x7ff3f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0004850 0x7ff3f0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff3f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080ce870 con 0x7ff3f0003a00 2023-07-04T18:51:22.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40815ea30 con 0x7ff3f0004850 2023-07-04T18:51:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f000fea0 con 0x7ff40815dfb0 2023-07-04T18:51:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.621+0000 7ff40659c700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f0003a00 0x7ff3f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff405d9b700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0004850 0x7ff3f0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff406d9d700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff3f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 -- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 msgr2=0x7ff3f0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff40815dfb0 0x7ff3f0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 -- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0004850 msgr2=0x7ff3f0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0004850 0x7ff3f0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 -- 172.21.15.77:0/1245929784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f00035b0 con 0x7ff3f0003a00 2023-07-04T18:51:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40659c700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f0003a00 0x7ff3f0003e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b60 tx=0x7ff3f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78dc7009bd497b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800e6c0 con 0x7ff3f0003a00 2023-07-04T18:51:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800e820 con 0x7ff3f0003a00 2023-07-04T18:51:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40559a700 1 -- 172.21.15.77:0/1245929784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800eb90 con 0x7ff3f0003a00 2023-07-04T18:51:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40cee8700 1 -- 172.21.15.77:0/1245929784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f0003a00 msgr2=0x7ff3f0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff40cee8700 1 --2- 172.21.15.77:0/1245929784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f0003a00 0x7ff3f0003e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8009b60 tx=0x7ff3f0005880 comp rx=0 tx=0).stop 2023-07-04T18:51:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.622+0000 7ff406d9d700 1 -- 172.21.15.77:0/1245929784 reap_dead start 2023-07-04T18:51:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.623+0000 7ff40cee8700 1 -- 172.21.15.77:0/1245929784 shutdown_connections 2023-07-04T18:51:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.623+0000 7ff40cee8700 1 -- 172.21.15.77:0/1245929784 >> 172.21.15.77:0/1245929784 conn(0x7ff4080b0d90 msgr2=0x7ff4080b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.623+0000 7ff40cee8700 1 -- 172.21.15.77:0/1245929784 shutdown_connections 2023-07-04T18:51:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.623+0000 7ff40cee8700 1 -- 172.21.15.77:0/1245929784 wait complete. 2023-07-04T18:51:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.624+0000 7ff40cee8700 1 Processor -- start 2023-07-04T18:51:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.624+0000 7ff40cee8700 1 -- start start 2023-07-04T18:51:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0003a00 0x7ff408153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4080ca080 0x7ff40815c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080cb240 con 0x7ff40814bc90 2023-07-04T18:51:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40814d2a0 con 0x7ff3f0003a00 2023-07-04T18:51:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40814d560 con 0x7ff4080ca080 2023-07-04T18:51:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43836/0 (socket says 172.21.15.77:43836) 2023-07-04T18:51:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 -- 172.21.15.77:0/2873496811 learned_addr learned my addr 172.21.15.77:0/2873496811 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40659c700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0003a00 0x7ff408153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff406d9d700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4080ca080 0x7ff40815c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 -- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4080ca080 msgr2=0x7ff40815c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4080ca080 0x7ff40815c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 -- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0003a00 msgr2=0x7ff408153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0003a00 0x7ff408153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 -- 172.21.15.77:0/2873496811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080caed0 con 0x7ff40814bc90 2023-07-04T18:51:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff405d9b700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc011930 tx=0x7ff3fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc00f750 con 0x7ff40814bc90 2023-07-04T18:51:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc00f8b0 con 0x7ff40814bc90 2023-07-04T18:51:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40814f2d0 con 0x7ff40814bc90 2023-07-04T18:51:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc00fc40 con 0x7ff40814bc90 2023-07-04T18:51:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.625+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40814c100 con 0x7ff40814bc90 2023-07-04T18:51:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.626+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4080cb3b0 con 0x7ff40814bc90 2023-07-04T18:51:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.627+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff3fc01d400 con 0x7ff40814bc90 2023-07-04T18:51:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.627+0000 7ff3f6ffd700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 0x7ff3e806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.627+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff3fc089a10 con 0x7ff40814bc90 2023-07-04T18:51:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.627+0000 7ff40659c700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 0x7ff3e806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.628+0000 7ff40659c700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 0x7ff3e806d1c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80090b0 tx=0x7ff3f801f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.632+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3fc054ac0 con 0x7ff40814bc90 2023-07-04T18:51:22.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.818+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4080cb5d0 con 0x7ff40814bc90 2023-07-04T18:51:22.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.819+0000 7ff3f6ffd700 1 -- 172.21.15.77:0/2873496811 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+18256 (secure 0 0 0) 0x7ff3fc0580e0 con 0x7ff40814bc90 2023-07-04T18:51:22.822 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:22.822 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:21.845114+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:22.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 msgr2=0x7ff3e806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 0x7ff3e806d1c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80090b0 tx=0x7ff3f801f040 comp rx=0 tx=0).stop 2023-07-04T18:51:22.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 msgr2=0x7ff40814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc011930 tx=0x7ff3fc011c40 comp rx=0 tx=0).stop 2023-07-04T18:51:22.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 shutdown_connections 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.821+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff3e806ad10 0x7ff3e806d1c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff4080ca080 0x7ff40815c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3f0003a00 0x7ff408153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 --2- 172.21.15.77:0/2873496811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff40814bc90 0x7ff40814ed90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 >> 172.21.15.77:0/2873496811 conn(0x7ff4080b0d90 msgr2=0x7ff40806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:22.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 shutdown_connections 2023-07-04T18:51:22.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.822+0000 7ff40cee8700 1 -- 172.21.15.77:0/2873496811 wait complete. 2023-07-04T18:51:22.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.956+0000 7fb4853bc700 1 Processor -- start 2023-07-04T18:51:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.956+0000 7fb4853bc700 1 -- start start 2023-07-04T18:51:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb4853bc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb480159280 0x7fb48006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb4853bc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48006d890 con 0x7fb48006c770 2023-07-04T18:51:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb4853bc700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48006cc80 con 0x7fb4800cdca0 2023-07-04T18:51:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb4853bc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48006cf60 con 0x7fb480159280 2023-07-04T18:51:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb47effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb480159280 0x7fb48006c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb47e7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb48006c770 0x7fb4800cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:50982/0 (socket says 172.21.15.77:50982) 2023-07-04T18:51:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.957+0000 7fb47e7fc700 1 -- 172.21.15.77:0/1765033535 learned_addr learned my addr 172.21.15.77:0/1765033535 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:22.961 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47effd700 1 -- 172.21.15.77:0/1765033535 >> v1:172.21.15.82:6789/0 conn(0x7fb4800cdca0 legacy=0x7fb4800ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47effd700 1 -- 172.21.15.77:0/1765033535 >> v1:172.21.15.77:6789/0 conn(0x7fb48006c770 legacy=0x7fb4800cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47dffb700 1 -- 172.21.15.77:0/1765033535 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542187699 0 0) 0x7fb48006d890 con 0x7fb48006c770 2023-07-04T18:51:22.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47effd700 1 -- 172.21.15.77:0/1765033535 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48006d240 con 0x7fb480159280 2023-07-04T18:51:22.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47effd700 1 --2- 172.21.15.77:0/1765033535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb480159280 0x7fb48006c230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb474009840 tx=0x7fb474009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82748957d7f9cb26 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.962 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47dffb700 1 -- 172.21.15.77:0/1765033535 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474011830 con 0x7fb480159280 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47dffb700 1 -- 172.21.15.77:0/1765033535 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb474011990 con 0x7fb480159280 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb47dffb700 1 -- 172.21.15.77:0/1765033535 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474011d20 con 0x7fb480159280 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb4853bc700 1 -- 172.21.15.77:0/1765033535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb480159280 msgr2=0x7fb48006c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1765033535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb480159280 0x7fb48006c230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb474009840 tx=0x7fb474009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb4853bc700 1 -- 172.21.15.77:0/1765033535 shutdown_connections 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.958+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1765033535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb480159280 0x7fb48006c230 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.963 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.959+0000 7fb4853bc700 1 -- 172.21.15.77:0/1765033535 >> 172.21.15.77:0/1765033535 conn(0x7fb48006bba0 msgr2=0x7fb4800b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:22.964 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.959+0000 7fb4853bc700 1 -- 172.21.15.77:0/1765033535 shutdown_connections 2023-07-04T18:51:22.964 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.959+0000 7fb4853bc700 1 -- 172.21.15.77:0/1765033535 wait complete. 2023-07-04T18:51:22.964 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.960+0000 7fb4853bc700 1 Processor -- start 2023-07-04T18:51:22.964 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.960+0000 7fb4853bc700 1 -- start start 2023-07-04T18:51:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4800cdca0 0x7fb4800c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb480159280 0x7fb480069e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800c6860 con 0x7fb48006c770 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb480159fc0 con 0x7fb4800cdca0 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4801f1fc0 con 0x7fb480159280 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43848/0 (socket says 172.21.15.77:43848) 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 -- 172.21.15.77:0/1260690827 learned_addr learned my addr 172.21.15.77:0/1260690827 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 -- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb480159280 msgr2=0x7fb480069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47e7fc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4800cdca0 0x7fb4800c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb480159280 0x7fb480069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 -- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4800cdca0 msgr2=0x7fb4800c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4800cdca0 0x7fb4800c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 -- 172.21.15.77:0/1260690827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4740094a0 con 0x7fb48006c770 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb47effd700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4740090b0 tx=0x7fb474004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4740036a0 con 0x7fb48006c770 2023-07-04T18:51:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47401e070 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4800c5ce0 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb474003a20 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.961+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4800c62a0 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.962+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb480159d20 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.963+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb474003b80 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.963+0000 7fb4677fe700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 0x7fb46c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.963+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb47408a660 con 0x7fb48006c770 2023-07-04T18:51:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.963+0000 7fb47e7fc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 0x7fb46c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.964+0000 7fb47e7fc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 0x7fb46c06d1c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4800ae4d0 tx=0x7fb470009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:22.968+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb480159d20 con 0x7fb48006c770 2023-07-04T18:51:23.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.156+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb480159d20 con 0x7fb48006c770 2023-07-04T18:51:23.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.157+0000 7fb4677fe700 1 -- 172.21.15.77:0/1260690827 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+18256 (secure 0 0 0) 0x7fb480159d20 con 0x7fb48006c770 2023-07-04T18:51:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:23.161 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:21.845114+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.159+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 msgr2=0x7fb46c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.159+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 0x7fb46c06d1c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb4800ae4d0 tx=0x7fb470009040 comp rx=0 tx=0).stop 2023-07-04T18:51:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.159+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 msgr2=0x7fb480142d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.159+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4740090b0 tx=0x7fb474004750 comp rx=0 tx=0).stop 2023-07-04T18:51:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 shutdown_connections 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb46c06ad10 0x7fb46c06d1c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb48006c770 0x7fb480142d90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb480159280 0x7fb480069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 --2- 172.21.15.77:0/1260690827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb4800cdca0 0x7fb4800c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 >> 172.21.15.77:0/1260690827 conn(0x7fb48006bba0 msgr2=0x7fb48015cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 shutdown_connections 2023-07-04T18:51:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.160+0000 7fb4853bc700 1 -- 172.21.15.77:0/1260690827 wait complete. 2023-07-04T18:51:23.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-04T18:51:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.298+0000 7f7fc1e29700 1 Processor -- start 2023-07-04T18:51:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.298+0000 7f7fc1e29700 1 -- start start 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fc1e29700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fbc1440b0 0x7f7fbc15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fc1e29700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fc1e29700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fbc15f180 con 0x7f7fbc0d0000 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fc1e29700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fbc0d0480 con 0x7f7fbc1440b0 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fc1e29700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc0cee30 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fbc1440b0 0x7f7fbc15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7fbc1440b0 0x7f7fbc15ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43856/0 (socket says 172.21.15.77:43856) 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/28487754 learned_addr learned my addr 172.21.15.77:0/28487754 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbaffd700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/28487754 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0ce8c0 msgr2=0x7f7fbc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/28487754 >> v1:172.21.15.82:6789/0 conn(0x7f7fbc0d0000 legacy=0x7f7fbc0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/28487754 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0cf110 con 0x7f7fbc1440b0 2023-07-04T18:51:23.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fbb7fe700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc15ec40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00098f0 tx=0x7f7fb0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef938863bf9fb76 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:23.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fba7fc700 1 -- 172.21.15.77:0/28487754 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000b8a0 con 0x7f7fbc1440b0 2023-07-04T18:51:23.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fba7fc700 1 -- 172.21.15.77:0/28487754 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb000ba00 con 0x7f7fbc1440b0 2023-07-04T18:51:23.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.299+0000 7f7fba7fc700 1 -- 172.21.15.77:0/28487754 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000bd90 con 0x7f7fbc1440b0 2023-07-04T18:51:23.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 -- 172.21.15.77:0/28487754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc1440b0 msgr2=0x7f7fbc15ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc15ec40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00098f0 tx=0x7f7fb0009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:23.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 -- 172.21.15.77:0/28487754 shutdown_connections 2023-07-04T18:51:23.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc15ec40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/28487754 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 -- 172.21.15.77:0/28487754 >> 172.21.15.77:0/28487754 conn(0x7f7fbc06d900 msgr2=0x7f7fbc06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:23.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 -- 172.21.15.77:0/28487754 shutdown_connections 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.300+0000 7f7fc1e29700 1 -- 172.21.15.77:0/28487754 wait complete. 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.301+0000 7f7fc1e29700 1 Processor -- start 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.301+0000 7f7fc1e29700 1 -- start start 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0d0000 0x7f7fbc06b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc1f3540 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc06bea0 con 0x7f7fbc1440b0 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc06c170 con 0x7f7fbc0d0000 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43862/0 (socket says 172.21.15.77:43862) 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/3735193363 learned_addr learned my addr 172.21.15.77:0/3735193363 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:23.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0d0000 msgr2=0x7f7fbc06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0d0000 0x7f7fbc06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fbc1440b0 msgr2=0x7f7fbc0c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 -- 172.21.15.77:0/3735193363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb0009550 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fbb7fe700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009c40 tx=0x7f7fb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00045d0 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb001f070 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc06c450 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00038e0 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.303+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fbc0c8f80 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.304+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fbc1f3680 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.305+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7fb0003a40 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.305+0000 7f7fa3fff700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 0x7f7fa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:23.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.305+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7fb0021070 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.305+0000 7f7fbaffd700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 0x7f7fa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:23.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.306+0000 7f7fbaffd700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 0x7f7fa806d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc1f6770 tx=0x7f7fac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:23.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.310+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7fbc1f3680 con 0x7f7fbc0ce8c0 2023-07-04T18:51:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:23.542+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7fbc06c970 con 0x7f7fbc0ce8c0 2023-07-04T18:51:24.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.490+0000 7f7fa3fff700 1 -- 172.21.15.77:0/3735193363 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7fb0055e50 con 0x7f7fbc0ce8c0 2023-07-04T18:51:24.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.492+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 msgr2=0x7f7fa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.492+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 0x7f7fa806d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc1f6770 tx=0x7f7fac008040 comp rx=0 tx=0).stop 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.492+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 msgr2=0x7f7fbc0cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.492+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009c40 tx=0x7f7fb0004750 comp rx=0 tx=0).stop 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 shutdown_connections 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7fa806ace0 0x7f7fa806d190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fbc0ce8c0 0x7f7fbc0cacd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fbc1440b0 0x7f7fbc0c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 --2- 172.21.15.77:0/3735193363 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7fbc0d0000 0x7f7fbc06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 >> 172.21.15.77:0/3735193363 conn(0x7f7fbc06d900 msgr2=0x7f7fbc0b4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:24.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 shutdown_connections 2023-07-04T18:51:24.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.493+0000 7f7fc1e29700 1 -- 172.21.15.77:0/3735193363 wait complete. 2023-07-04T18:51:24.509 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-07-04T18:51:24.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-07-04T18:51:24.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.637+0000 7fb1272aa700 1 Processor -- start 2023-07-04T18:51:24.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.637+0000 7fb1272aa700 1 -- start start 2023-07-04T18:51:24.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb1272aa700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 0x7fb12015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb1272aa700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb12006a550 0x7fb12015e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb1272aa700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1201582f0 con 0x7fb12015e580 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb1272aa700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12015de50 con 0x7fb12006a550 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb1272aa700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12015b0b0 con 0x7fb120157f20 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb125847700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb12006a550 0x7fb12015e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb125847700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb12006a550 0x7fb12015e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43872/0 (socket says 172.21.15.77:43872) 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.638+0000 7fb125847700 1 -- 172.21.15.77:0/1025851091 learned_addr learned my addr 172.21.15.77:0/1025851091 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125046700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 0x7fb12015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125847700 1 -- 172.21.15.77:0/1025851091 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 msgr2=0x7fb12015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125847700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 0x7fb12015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125847700 1 -- 172.21.15.77:0/1025851091 >> v1:172.21.15.82:6789/0 conn(0x7fb12015e580 legacy=0x7fb120069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:24.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125847700 1 -- 172.21.15.77:0/1025851091 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12015b390 con 0x7fb12006a550 2023-07-04T18:51:24.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125046700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 0x7fb12015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:24.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb125847700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12015e9f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb118009e50 tx=0x7fb11800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d06582dc31d785 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:24.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb117fff700 1 -- 172.21.15.77:0/1025851091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11800e070 con 0x7fb12006a550 2023-07-04T18:51:24.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb117fff700 1 -- 172.21.15.77:0/1025851091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118003a20 con 0x7fb12006a550 2023-07-04T18:51:24.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb117fff700 1 -- 172.21.15.77:0/1025851091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118003dd0 con 0x7fb12006a550 2023-07-04T18:51:24.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb1272aa700 1 -- 172.21.15.77:0/1025851091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 msgr2=0x7fb12015e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:24.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.639+0000 7fb1272aa700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12015e9f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb118009e50 tx=0x7fb11800b330 comp rx=0 tx=0).stop 2023-07-04T18:51:24.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 -- 172.21.15.77:0/1025851091 shutdown_connections 2023-07-04T18:51:24.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12015e9f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 --2- 172.21.15.77:0/1025851091 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb120157f20 0x7fb12015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 -- 172.21.15.77:0/1025851091 >> 172.21.15.77:0/1025851091 conn(0x7fb12006af50 msgr2=0x7fb12006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:24.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 -- 172.21.15.77:0/1025851091 shutdown_connections 2023-07-04T18:51:24.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.640+0000 7fb1272aa700 1 -- 172.21.15.77:0/1025851091 wait complete. 2023-07-04T18:51:24.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 Processor -- start 2023-07-04T18:51:24.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 -- start start 2023-07-04T18:51:24.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb120157f20 0x7fb1200b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb12015e580 0x7fb12006c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12014f9e0 con 0x7fb12006a550 2023-07-04T18:51:24.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.641+0000 7fb1272aa700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200b3f20 con 0x7fb120157f20 2023-07-04T18:51:24.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb1272aa700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200b3310 con 0x7fb12015e580 2023-07-04T18:51:24.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43874/0 (socket says 172.21.15.77:43874) 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 -- 172.21.15.77:0/452436699 learned_addr learned my addr 172.21.15.77:0/452436699 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb124845700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb120157f20 0x7fb1200b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125847700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb12015e580 0x7fb12006c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 -- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb12015e580 msgr2=0x7fb12006c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:24.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb12015e580 0x7fb12006c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 -- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb120157f20 msgr2=0x7fb1200b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb120157f20 0x7fb1200b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 -- 172.21.15.77:0/452436699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118009b30 con 0x7fb12006a550 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb125046700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c750 tx=0x7fb11000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110011800 con 0x7fb12006a550 2023-07-04T18:51:24.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb110011960 con 0x7fb12006a550 2023-07-04T18:51:24.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1200b35f0 con 0x7fb12006a550 2023-07-04T18:51:24.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110011cd0 con 0x7fb12006a550 2023-07-04T18:51:24.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.642+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1200b3bb0 con 0x7fb12006a550 2023-07-04T18:51:24.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.643+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12004e210 con 0x7fb12006a550 2023-07-04T18:51:24.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.644+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb110021350 con 0x7fb12006a550 2023-07-04T18:51:24.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.644+0000 7fb115ffb700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 0x7fb10406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:24.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.644+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb110089310 con 0x7fb12006a550 2023-07-04T18:51:24.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.644+0000 7fb124845700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 0x7fb10406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:24.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.645+0000 7fb124845700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 0x7fb10406d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1200af280 tx=0x7fb11c0090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.648+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb110057dc0 con 0x7fb12006a550 2023-07-04T18:51:24.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:24.882+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb1200d0180 con 0x7fb12006a550 2023-07-04T18:51:25.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.495+0000 7fb115ffb700 1 -- 172.21.15.77:0/452436699 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb110057950 con 0x7fb12006a550 2023-07-04T18:51:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 msgr2=0x7fb10406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 0x7fb10406d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1200af280 tx=0x7fb11c0090e0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 msgr2=0x7fb12014ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c750 tx=0x7fb11000ca60 comp rx=0 tx=0).stop 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 shutdown_connections 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb10406ace0 0x7fb10406d190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb12006a550 0x7fb12014ee00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb12015e580 0x7fb12006c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 --2- 172.21.15.77:0/452436699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb120157f20 0x7fb1200b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 >> 172.21.15.77:0/452436699 conn(0x7fb12006af50 msgr2=0x7fb1200cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:25.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 shutdown_connections 2023-07-04T18:51:25.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.498+0000 7fb1272aa700 1 -- 172.21.15.77:0/452436699 wait complete. 2023-07-04T18:51:25.515 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-07-04T18:51:25.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:51:25.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 Processor -- start 2023-07-04T18:51:25.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 -- start start 2023-07-04T18:51:25.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdb806c310 0x7efdb815df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb815e8b0 con 0x7efdb815a410 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb806c7b0 con 0x7efdb806c310 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.640+0000 7efdbedbb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb815a7e0 con 0x7efdb815e470 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdb806c310 0x7efdb815df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:25.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdbcb57700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efdb815a410 0x7efdb80ca7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51020/0 (socket says 172.21.15.77:51020) 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdb806c310 0x7efdb815df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43884/0 (socket says 172.21.15.77:43884) 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdbcb57700 1 -- 172.21.15.77:0/416187746 learned_addr learned my addr 172.21.15.77:0/416187746 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 -- 172.21.15.77:0/416187746 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 msgr2=0x7efdb806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 --2- 172.21.15.77:0/416187746 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 -- 172.21.15.77:0/416187746 >> v1:172.21.15.77:6789/0 conn(0x7efdb815a410 legacy=0x7efdb80ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 -- 172.21.15.77:0/416187746 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb80cb570 con 0x7efdb806c310 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb77fe700 1 -- 172.21.15.77:0/416187746 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532948552 0 0) 0x7efdb815e8b0 con 0x7efdb815a410 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb7fff700 1 --2- 172.21.15.77:0/416187746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb815df30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efda4009840 tx=0x7efda4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d4d8b5846ce4b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb77fe700 1 -- 172.21.15.77:0/416187746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4011830 con 0x7efdb806c310 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb77fe700 1 -- 172.21.15.77:0/416187746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda4011990 con 0x7efdb806c310 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.641+0000 7efdb77fe700 1 -- 172.21.15.77:0/416187746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4011d20 con 0x7efdb806c310 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 -- 172.21.15.77:0/416187746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 msgr2=0x7efdb815df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 --2- 172.21.15.77:0/416187746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb815df30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efda4009840 tx=0x7efda4009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 -- 172.21.15.77:0/416187746 shutdown_connections 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 --2- 172.21.15.77:0/416187746 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 --2- 172.21.15.77:0/416187746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb815df30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 -- 172.21.15.77:0/416187746 >> 172.21.15.77:0/416187746 conn(0x7efdb80cf220 msgr2=0x7efdb80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 -- 172.21.15.77:0/416187746 shutdown_connections 2023-07-04T18:51:25.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.642+0000 7efdbedbb700 1 -- 172.21.15.77:0/416187746 wait complete. 2023-07-04T18:51:25.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.643+0000 7efdbedbb700 1 Processor -- start 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.643+0000 7efdbedbb700 1 -- start start 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 0x7efdb80b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb80b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8155b20 con 0x7efdb806c310 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb809e150 con 0x7efdb815a410 2023-07-04T18:51:25.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb809e430 con 0x7efdb815e470 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43890/0 (socket says 172.21.15.77:43890) 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 -- 172.21.15.77:0/808146884 learned_addr learned my addr 172.21.15.77:0/808146884 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbd358700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb80b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdb7fff700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 0x7efdb80b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 -- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 msgr2=0x7efdb80b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb80b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 -- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 msgr2=0x7efdb80b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 0x7efdb80b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 -- 172.21.15.77:0/808146884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40094a0 con 0x7efdb806c310 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdb7fff700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 0x7efdb80b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbcb57700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdac0011e0 tx=0x7efdac00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:25.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac004c80 con 0x7efdb806c310 2023-07-04T18:51:25.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdac004e00 con 0x7efdb806c310 2023-07-04T18:51:25.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb809e710 con 0x7efdb806c310 2023-07-04T18:51:25.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.644+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac007310 con 0x7efdb806c310 2023-07-04T18:51:25.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.645+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb809ecd0 con 0x7efdb806c310 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.645+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb804e210 con 0x7efdb806c310 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.646+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efdac007470 con 0x7efdb806c310 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.646+0000 7efdb57fa700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 0x7efd9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.646+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efdac089200 con 0x7efdb806c310 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.646+0000 7efdb7fff700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 0x7efd9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:25.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.647+0000 7efdb7fff700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 0x7efd9c06d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efdb8156fc0 tx=0x7efda4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:25.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.651+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efdac0542b0 con 0x7efdb806c310 2023-07-04T18:51:25.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.879+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdb80b51a0 con 0x7efdb806c310 2023-07-04T18:51:25.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.879+0000 7efdb57fa700 1 -- 172.21.15.77:0/808146884 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7efdac0578d0 con 0x7efdb806c310 2023-07-04T18:51:25.881 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:25.881 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:51:25.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 msgr2=0x7efd9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 0x7efd9c06d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efdb8156fc0 tx=0x7efda4008030 comp rx=0 tx=0).stop 2023-07-04T18:51:25.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 msgr2=0x7efdb80ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:25.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efdac0011e0 tx=0x7efdac00cae0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 shutdown_connections 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd9c06ace0 0x7efd9c06d190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdb806c310 0x7efdb80ae500 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdb815e470 0x7efdb80b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 --2- 172.21.15.77:0/808146884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb815a410 0x7efdb80b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 >> 172.21.15.77:0/808146884 conn(0x7efdb80cf220 msgr2=0x7efdb806d260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 shutdown_connections 2023-07-04T18:51:25.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:25.882+0000 7efdbedbb700 1 -- 172.21.15.77:0/808146884 wait complete. 2023-07-04T18:51:25.885 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:51:25.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:26.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 Processor -- start 2023-07-04T18:51:26.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 -- start start 2023-07-04T18:51:26.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f539015c640 0x7f53900cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 0x7f5390070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53900717f0 con 0x7f5390070db0 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5390071250 con 0x7f539015c640 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f5395e02700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900cd290 con 0x7f539015ca10 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f538f7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f539015c640 0x7f53900cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f538f7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f539015c640 0x7f53900cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43896/0 (socket says 172.21.15.77:43896) 2023-07-04T18:51:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.014+0000 7f538f7fe700 1 -- 172.21.15.77:0/926696549 learned_addr learned my addr 172.21.15.77:0/926696549 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:26.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538effd700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 0x7f5390070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538f7fe700 1 -- 172.21.15.77:0/926696549 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 msgr2=0x7f5390070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:26.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538f7fe700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 0x7f5390070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538f7fe700 1 -- 172.21.15.77:0/926696549 >> v1:172.21.15.82:6789/0 conn(0x7f5390070db0 legacy=0x7f53900cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538f7fe700 1 -- 172.21.15.77:0/926696549 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900ccfb0 con 0x7f539015c640 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538effd700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 0x7f5390070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538f7fe700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015c640 0x7f53900cf120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53800098f0 tx=0x7f5380009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665cf68f290f791 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538e7fc700 1 -- 172.21.15.77:0/926696549 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000b8a0 con 0x7f539015c640 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538e7fc700 1 -- 172.21.15.77:0/926696549 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538000ba00 con 0x7f539015c640 2023-07-04T18:51:26.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.015+0000 7f538e7fc700 1 -- 172.21.15.77:0/926696549 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000bd90 con 0x7f539015c640 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 -- 172.21.15.77:0/926696549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015c640 msgr2=0x7f53900cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015c640 0x7f53900cf120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53800098f0 tx=0x7f5380009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 -- 172.21.15.77:0/926696549 shutdown_connections 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015c640 0x7f53900cf120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 --2- 172.21.15.77:0/926696549 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f539015ca10 0x7f5390070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 -- 172.21.15.77:0/926696549 >> 172.21.15.77:0/926696549 conn(0x7f539006bba0 msgr2=0x7f53900b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 -- 172.21.15.77:0/926696549 shutdown_connections 2023-07-04T18:51:26.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.016+0000 7f5395e02700 1 -- 172.21.15.77:0/926696549 wait complete. 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.017+0000 7f5395e02700 1 Processor -- start 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.017+0000 7f5395e02700 1 -- start start 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5390070db0 0x7f539015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539015c640 0x7f53900b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53901f2020 con 0x7f539015ca10 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539015d0c0 con 0x7f539015c640 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f5395e02700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53900b3c20 con 0x7f5390070db0 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f538ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f538ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43906/0 (socket says 172.21.15.77:43906) 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.018+0000 7f538ffff700 1 -- 172.21.15.77:0/4188749460 learned_addr learned my addr 172.21.15.77:0/4188749460 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538effd700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539015c640 0x7f53900b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538f7fe700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5390070db0 0x7f539015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 -- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5390070db0 msgr2=0x7f539015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5390070db0 0x7f539015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 -- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539015c640 msgr2=0x7f53900b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539015c640 0x7f53900b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 -- 172.21.15.77:0/4188749460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5380009550 con 0x7f539015ca10 2023-07-04T18:51:26.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f538ffff700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f538800cb50 tx=0x7f538800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388003c00 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388003d60 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800ab70 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53900b3f00 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.019+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53900b44c0 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.020+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539015d230 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.020+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f538800acd0 con 0x7f539015ca10 2023-07-04T18:51:26.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.021+0000 7f5387fff700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 0x7f537006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.029 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.021+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5388089520 con 0x7f539015ca10 2023-07-04T18:51:26.029 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.021+0000 7f538f7fe700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 0x7f537006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.029 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.021+0000 7f538f7fe700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 0x7f537006d190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5380014040 tx=0x7f5380008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.029 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.026+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f539015d230 con 0x7f539015ca10 2023-07-04T18:51:26.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.211+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f539015d230 con 0x7f539015ca10 2023-07-04T18:51:26.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.212+0000 7f5387fff700 1 -- 172.21.15.77:0/4188749460 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f539015d230 con 0x7f539015ca10 2023-07-04T18:51:26.215 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:26.215 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:26.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 msgr2=0x7f537006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 0x7f537006d190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5380014040 tx=0x7f5380008030 comp rx=0 tx=0).stop 2023-07-04T18:51:26.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 msgr2=0x7f53901e6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f538800cb50 tx=0x7f538800cf10 comp rx=0 tx=0).stop 2023-07-04T18:51:26.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 shutdown_connections 2023-07-04T18:51:26.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f537006ace0 0x7f537006d190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f539015ca10 0x7f53901e6080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5390070db0 0x7f539015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 --2- 172.21.15.77:0/4188749460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539015c640 0x7f53900b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 >> 172.21.15.77:0/4188749460 conn(0x7f539006bba0 msgr2=0x7f53900cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 shutdown_connections 2023-07-04T18:51:26.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.214+0000 7f5395e02700 1 -- 172.21.15.77:0/4188749460 wait complete. 2023-07-04T18:51:26.228 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-04T18:51:26.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:51:26.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 Processor -- start 2023-07-04T18:51:26.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 -- start start 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f5fcc158c90 0x7f5fcc0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc071ad0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fcc0cb9f0 con 0x7f5fcc158c90 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fd0c50700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0c9fd0 con 0x7f5fcc071670 2023-07-04T18:51:26.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fc9d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5fcc0ce400 0x7f5fcc070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51050/0 (socket says 172.21.15.77:51050) 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.347+0000 7f5fc9d9b700 1 -- 172.21.15.77:0/285599012 learned_addr learned my addr 172.21.15.77:0/285599012 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 --2- 172.21.15.77:0/285599012 >> v2:172.21.15.82:3300/0 conn(0x7f5fcc158c90 0x7f5fcc0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fcad9d700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197193001 0 0) 0x7f5fcc071ad0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb40046c0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 -- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 msgr2=0x7f5fcc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127534626 0 0) 0x7f5fb40046c0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc0ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 -- 172.21.15.77:0/285599012 >> v1:172.21.15.77:6789/0 conn(0x7f5fcc0ce400 legacy=0x7f5fcc070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 -- 172.21.15.77:0/285599012 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc071d50 con 0x7f5fcc158c90 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fca59c700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc0ca3f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009840 tx=0x7f5fbc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f5b7fd5e2ddd7ad server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0001670 con 0x7f5fcc158c90 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc004030 con 0x7f5fcc158c90 2023-07-04T18:51:26.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.348+0000 7f5fc959a700 1 -- 172.21.15.77:0/285599012 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0043a0 con 0x7f5fcc158c90 2023-07-04T18:51:26.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 -- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 msgr2=0x7f5fcc0ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc0ca3f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009840 tx=0x7f5fbc009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 -- 172.21.15.77:0/285599012 shutdown_connections 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/285599012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc0ca3f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 -- 172.21.15.77:0/285599012 >> 172.21.15.77:0/285599012 conn(0x7f5fcc0cf220 msgr2=0x7f5fcc0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 -- 172.21.15.77:0/285599012 shutdown_connections 2023-07-04T18:51:26.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.349+0000 7f5fd0c50700 1 -- 172.21.15.77:0/285599012 wait complete. 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.350+0000 7f5fd0c50700 1 Processor -- start 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.350+0000 7f5fd0c50700 1 -- start start 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc1efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc14beb0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc15c080 con 0x7f5fcc158c90 2023-07-04T18:51:26.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fd0c50700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0c5cc0 con 0x7f5fcc071670 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43918/0 (socket says 172.21.15.77:43918) 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 -- 172.21.15.77:0/1509287059 learned_addr learned my addr 172.21.15.77:0/1509287059 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fca59c700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc1efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 -- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 msgr2=0x7f5fcc1efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fcad9d700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc1efc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 -- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 msgr2=0x7f5fcc06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.351+0000 7f5fc9d9b700 1 -- 172.21.15.77:0/1509287059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0094a0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fc9d9b700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00011e0 tx=0x7f5fc000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0007560 con 0x7f5fcc0ce400 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc00076c0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc15bac0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0004a90 con 0x7f5fcc0ce400 2023-07-04T18:51:26.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.352+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc14bad0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.353+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc14b930 con 0x7f5fcc0ce400 2023-07-04T18:51:26.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.353+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5fc0004bf0 con 0x7f5fcc0ce400 2023-07-04T18:51:26.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.354+0000 7f5fbaffd700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 0x7f5fac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.354+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5fc0089190 con 0x7f5fcc0ce400 2023-07-04T18:51:26.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.354+0000 7f5fca59c700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 0x7f5fac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.354+0000 7f5fca59c700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 0x7f5fac06d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000c00 tx=0x7f5fbc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.358+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5fcc14b930 con 0x7f5fcc0ce400 2023-07-04T18:51:26.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.544+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fcc0c9910 con 0x7f5fac06ace0 2023-07-04T18:51:26.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.553+0000 7f5fbaffd700 1 -- 172.21.15.77:0/1509287059 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286694 (secure 0 0 0) 0x7f5fbc004320 con 0x7f5fac06ace0 2023-07-04T18:51:26.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:26.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 msgr2=0x7f5fac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 0x7f5fac06d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc000c00 tx=0x7f5fbc008030 comp rx=0 tx=0).stop 2023-07-04T18:51:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 msgr2=0x7f5fcc06c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00011e0 tx=0x7f5fc000cb00 comp rx=0 tx=0).stop 2023-07-04T18:51:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 shutdown_connections 2023-07-04T18:51:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5fac06ace0 0x7f5fac06d190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fcc158c90 0x7f5fcc06cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5fcc071670 0x7f5fcc1efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 --2- 172.21.15.77:0/1509287059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fcc0ce400 0x7f5fcc06c650 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 >> 172.21.15.77:0/1509287059 conn(0x7f5fcc0cf220 msgr2=0x7f5fcc06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 shutdown_connections 2023-07-04T18:51:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.556+0000 7f5fd0c50700 1 -- 172.21.15.77:0/1509287059 wait complete. 2023-07-04T18:51:26.562 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:51:26.569 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-07-04T18:51:25.176720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218668,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754756052,"statfs":{"total":773094113280,"available":772870197248,"internally_reserved":0,"allocated":3125248,"data_stored":1868987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.733296"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316037+0000","last_change":"2023-07-04T18:51:22.314047+0000","last_active":"2023-07-04T18:51:22.316037+0000","last_peered":"2023-07-04T18:51:22.316037+0000","last_clean":"2023-07-04T18:51:22.316037+0000","last_became_active":"2023-07-04T18:51:22.313772+0000","last_became_peered":"2023-07-04T18:51:22.313772+0000","last_unstale":"2023-07-04T18:51:22.316037+0000","last_undegraded":"2023-07-04T18:51:22.316037+0000","last_fullsized":"2023-07-04T18:51:22.316037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:07:21.587682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325987+0000","last_change":"2023-07-04T18:51:22.324407+0000","last_active":"2023-07-04T18:51:22.325987+0000","last_peered":"2023-07-04T18:51:22.325987+0000","last_clean":"2023-07-04T18:51:22.325987+0000","last_became_active":"2023-07-04T18:51:22.323862+0000","last_became_peered":"2023-07-04T18:51:22.323862+0000","last_unstale":"2023-07-04T18:51:22.325987+0000","last_undegraded":"2023-07-04T18:51:22.325987+0000","last_fullsized":"2023-07-04T18:51:22.325987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:46.737923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304530+0000","last_change":"2023-07-04T18:51:22.319081+0000","last_active":"2023-07-04T18:51:23.304530+0000","last_peered":"2023-07-04T18:51:23.304530+0000","last_clean":"2023-07-04T18:51:23.304530+0000","last_became_active":"2023-07-04T18:51:22.318806+0000","last_became_peered":"2023-07-04T18:51:22.318806+0000","last_unstale":"2023-07-04T18:51:23.304530+0000","last_undegraded":"2023-07-04T18:51:23.304530+0000","last_fullsized":"2023-07-04T18:51:23.304530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303285+0000","last_change":"2023-07-04T18:51:22.319728+0000","last_active":"2023-07-04T18:51:23.303285+0000","last_peered":"2023-07-04T18:51:23.303285+0000","last_clean":"2023-07-04T18:51:23.303285+0000","last_became_active":"2023-07-04T18:51:22.319417+0000","last_became_peered":"2023-07-04T18:51:22.319417+0000","last_unstale":"2023-07-04T18:51:23.303285+0000","last_undegraded":"2023-07-04T18:51:23.303285+0000","last_fullsized":"2023-07-04T18:51:23.303285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320125+0000","last_change":"2023-07-04T18:51:22.317945+0000","last_active":"2023-07-04T18:51:22.320125+0000","last_peered":"2023-07-04T18:51:22.320125+0000","last_clean":"2023-07-04T18:51:22.320125+0000","last_became_active":"2023-07-04T18:51:22.317635+0000","last_became_peered":"2023-07-04T18:51:22.317635+0000","last_unstale":"2023-07-04T18:51:22.320125+0000","last_undegraded":"2023-07-04T18:51:22.320125+0000","last_fullsized":"2023-07-04T18:51:22.320125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:19:18.779329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304259+0000","last_change":"2023-07-04T18:51:22.320918+0000","last_active":"2023-07-04T18:51:23.304259+0000","last_peered":"2023-07-04T18:51:23.304259+0000","last_clean":"2023-07-04T18:51:23.304259+0000","last_became_active":"2023-07-04T18:51:22.320502+0000","last_became_peered":"2023-07-04T18:51:22.320502+0000","last_unstale":"2023-07-04T18:51:23.304259+0000","last_undegraded":"2023-07-04T18:51:23.304259+0000","last_fullsized":"2023-07-04T18:51:23.304259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450875+0000","last_change":"2023-07-04T18:51:22.323607+0000","last_active":"2023-07-04T18:51:23.450875+0000","last_peered":"2023-07-04T18:51:23.450875+0000","last_clean":"2023-07-04T18:51:23.450875+0000","last_became_active":"2023-07-04T18:51:22.322940+0000","last_became_peered":"2023-07-04T18:51:22.322940+0000","last_unstale":"2023-07-04T18:51:23.450875+0000","last_undegraded":"2023-07-04T18:51:23.450875+0000","last_fullsized":"2023-07-04T18:51:23.450875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325268+0000","last_change":"2023-07-04T18:51:22.321665+0000","last_active":"2023-07-04T18:51:22.325268+0000","last_peered":"2023-07-04T18:51:22.325268+0000","last_clean":"2023-07-04T18:51:22.325268+0000","last_became_active":"2023-07-04T18:51:22.321426+0000","last_became_peered":"2023-07-04T18:51:22.321426+0000","last_unstale":"2023-07-04T18:51:22.325268+0000","last_undegraded":"2023-07-04T18:51:22.325268+0000","last_fullsized":"2023-07-04T18:51:22.325268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320229+0000","last_change":"2023-07-04T18:51:22.318535+0000","last_active":"2023-07-04T18:51:22.320229+0000","last_peered":"2023-07-04T18:51:22.320229+0000","last_clean":"2023-07-04T18:51:22.320229+0000","last_became_active":"2023-07-04T18:51:22.318160+0000","last_became_peered":"2023-07-04T18:51:22.318160+0000","last_unstale":"2023-07-04T18:51:22.320229+0000","last_undegraded":"2023-07-04T18:51:22.320229+0000","last_fullsized":"2023-07-04T18:51:22.320229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:44:26.146578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324837+0000","last_change":"2023-07-04T18:51:22.319854+0000","last_active":"2023-07-04T18:51:22.324837+0000","last_peered":"2023-07-04T18:51:22.324837+0000","last_clean":"2023-07-04T18:51:22.324837+0000","last_became_active":"2023-07-04T18:51:22.319581+0000","last_became_peered":"2023-07-04T18:51:22.319581+0000","last_unstale":"2023-07-04T18:51:22.324837+0000","last_undegraded":"2023-07-04T18:51:22.324837+0000","last_fullsized":"2023-07-04T18:51:22.324837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304943+0000","last_change":"2023-07-04T18:51:22.322222+0000","last_active":"2023-07-04T18:51:23.304943+0000","last_peered":"2023-07-04T18:51:23.304943+0000","last_clean":"2023-07-04T18:51:23.304943+0000","last_became_active":"2023-07-04T18:51:22.321944+0000","last_became_peered":"2023-07-04T18:51:22.321944+0000","last_unstale":"2023-07-04T18:51:23.304943+0000","last_undegraded":"2023-07-04T18:51:23.304943+0000","last_fullsized":"2023-07-04T18:51:23.304943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:53.251606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.322567+0000","last_change":"2023-07-04T18:51:22.318773+0000","last_active":"2023-07-04T18:51:22.322567+0000","last_peered":"2023-07-04T18:51:22.322567+0000","last_clean":"2023-07-04T18:51:22.322567+0000","last_became_active":"2023-07-04T18:51:22.318495+0000","last_became_peered":"2023-07-04T18:51:22.318495+0000","last_unstale":"2023-07-04T18:51:22.322567+0000","last_undegraded":"2023-07-04T18:51:22.322567+0000","last_fullsized":"2023-07-04T18:51:22.322567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:19:18.779329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315977+0000","last_change":"2023-07-04T18:51:22.313313+0000","last_active":"2023-07-04T18:51:22.315977+0000","last_peered":"2023-07-04T18:51:22.315977+0000","last_clean":"2023-07-04T18:51:22.315977+0000","last_became_active":"2023-07-04T18:51:22.313049+0000","last_became_peered":"2023-07-04T18:51:22.313049+0000","last_unstale":"2023-07-04T18:51:22.315977+0000","last_undegraded":"2023-07-04T18:51:22.315977+0000","last_fullsized":"2023-07-04T18:51:22.315977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:31:55.772208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325873+0000","last_change":"2023-07-04T18:51:22.322812+0000","last_active":"2023-07-04T18:51:22.325873+0000","last_peered":"2023-07-04T18:51:22.325873+0000","last_clean":"2023-07-04T18:51:22.325873+0000","last_became_active":"2023-07-04T18:51:22.318629+0000","last_became_peered":"2023-07-04T18:51:22.318629+0000","last_unstale":"2023-07-04T18:51:22.325873+0000","last_undegraded":"2023-07-04T18:51:22.325873+0000","last_fullsized":"2023-07-04T18:51:22.325873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450996+0000","last_change":"2023-07-04T18:51:22.323740+0000","last_active":"2023-07-04T18:51:23.450996+0000","last_peered":"2023-07-04T18:51:23.450996+0000","last_clean":"2023-07-04T18:51:23.450996+0000","last_became_active":"2023-07-04T18:51:22.323321+0000","last_became_peered":"2023-07-04T18:51:22.323321+0000","last_unstale":"2023-07-04T18:51:23.450996+0000","last_undegraded":"2023-07-04T18:51:23.450996+0000","last_fullsized":"2023-07-04T18:51:23.450996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:33:48.187398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304015+0000","last_change":"2023-07-04T18:51:22.318780+0000","last_active":"2023-07-04T18:51:23.304015+0000","last_peered":"2023-07-04T18:51:23.304015+0000","last_clean":"2023-07-04T18:51:23.304015+0000","last_became_active":"2023-07-04T18:51:22.318460+0000","last_became_peered":"2023-07-04T18:51:22.318460+0000","last_unstale":"2023-07-04T18:51:23.304015+0000","last_undegraded":"2023-07-04T18:51:23.304015+0000","last_fullsized":"2023-07-04T18:51:23.304015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320211+0000","last_change":"2023-07-04T18:51:22.318170+0000","last_active":"2023-07-04T18:51:22.320211+0000","last_peered":"2023-07-04T18:51:22.320211+0000","last_clean":"2023-07-04T18:51:22.320211+0000","last_became_active":"2023-07-04T18:51:22.317879+0000","last_became_peered":"2023-07-04T18:51:22.317879+0000","last_unstale":"2023-07-04T18:51:22.320211+0000","last_undegraded":"2023-07-04T18:51:22.320211+0000","last_fullsized":"2023-07-04T18:51:22.320211+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:07:21.587682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304493+0000","last_change":"2023-07-04T18:51:22.318559+0000","last_active":"2023-07-04T18:51:23.304493+0000","last_peered":"2023-07-04T18:51:23.304493+0000","last_clean":"2023-07-04T18:51:23.304493+0000","last_became_active":"2023-07-04T18:51:22.317739+0000","last_became_peered":"2023-07-04T18:51:22.317739+0000","last_unstale":"2023-07-04T18:51:23.304493+0000","last_undegraded":"2023-07-04T18:51:23.304493+0000","last_fullsized":"2023-07-04T18:51:23.304493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315900+0000","last_change":"2023-07-04T18:51:22.310959+0000","last_active":"2023-07-04T18:51:22.315900+0000","last_peered":"2023-07-04T18:51:22.315900+0000","last_clean":"2023-07-04T18:51:22.315900+0000","last_became_active":"2023-07-04T18:51:22.310668+0000","last_became_peered":"2023-07-04T18:51:22.310668+0000","last_unstale":"2023-07-04T18:51:22.315900+0000","last_undegraded":"2023-07-04T18:51:22.315900+0000","last_fullsized":"2023-07-04T18:51:22.315900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:44:26.146578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304398+0000","last_change":"2023-07-04T18:51:22.318672+0000","last_active":"2023-07-04T18:51:23.304398+0000","last_peered":"2023-07-04T18:51:23.304398+0000","last_clean":"2023-07-04T18:51:23.304398+0000","last_became_active":"2023-07-04T18:51:22.317856+0000","last_became_peered":"2023-07-04T18:51:22.317856+0000","last_unstale":"2023-07-04T18:51:23.304398+0000","last_undegraded":"2023-07-04T18:51:23.304398+0000","last_fullsized":"2023-07-04T18:51:23.304398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451144+0000","last_change":"2023-07-04T18:51:22.321788+0000","last_active":"2023-07-04T18:51:23.451144+0000","last_peered":"2023-07-04T18:51:23.451144+0000","last_clean":"2023-07-04T18:51:23.451144+0000","last_became_active":"2023-07-04T18:51:22.321394+0000","last_became_peered":"2023-07-04T18:51:22.321394+0000","last_unstale":"2023-07-04T18:51:23.451144+0000","last_undegraded":"2023-07-04T18:51:23.451144+0000","last_fullsized":"2023-07-04T18:51:23.451144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:00:25.437637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.323983+0000","last_change":"2023-07-04T18:51:22.321742+0000","last_active":"2023-07-04T18:51:22.323983+0000","last_peered":"2023-07-04T18:51:22.323983+0000","last_clean":"2023-07-04T18:51:22.323983+0000","last_became_active":"2023-07-04T18:51:22.321526+0000","last_became_peered":"2023-07-04T18:51:22.321526+0000","last_unstale":"2023-07-04T18:51:22.323983+0000","last_undegraded":"2023-07-04T18:51:22.323983+0000","last_fullsized":"2023-07-04T18:51:22.323983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:34:07.018645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303613+0000","last_change":"2023-07-04T18:51:22.321095+0000","last_active":"2023-07-04T18:51:23.303613+0000","last_peered":"2023-07-04T18:51:23.303613+0000","last_clean":"2023-07-04T18:51:23.303613+0000","last_became_active":"2023-07-04T18:51:22.320848+0000","last_became_peered":"2023-07-04T18:51:22.320848+0000","last_unstale":"2023-07-04T18:51:23.303613+0000","last_undegraded":"2023-07-04T18:51:23.303613+0000","last_fullsized":"2023-07-04T18:51:23.303613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450972+0000","last_change":"2023-07-04T18:51:22.322035+0000","last_active":"2023-07-04T18:51:23.450972+0000","last_peered":"2023-07-04T18:51:23.450972+0000","last_clean":"2023-07-04T18:51:23.450972+0000","last_became_active":"2023-07-04T18:51:22.321725+0000","last_became_peered":"2023-07-04T18:51:22.321725+0000","last_unstale":"2023-07-04T18:51:23.450972+0000","last_undegraded":"2023-07-04T18:51:23.450972+0000","last_fullsized":"2023-07-04T18:51:23.450972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417751+0000","last_change":"2023-07-04T18:51:22.311616+0000","last_active":"2023-07-04T18:51:23.417751+0000","last_peered":"2023-07-04T18:51:23.417751+0000","last_clean":"2023-07-04T18:51:23.417751+0000","last_became_active":"2023-07-04T18:51:22.311344+0000","last_became_peered":"2023-07-04T18:51:22.311344+0000","last_unstale":"2023-07-04T18:51:23.417751+0000","last_undegraded":"2023-07-04T18:51:23.417751+0000","last_fullsized":"2023-07-04T18:51:23.417751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417648+0000","last_change":"2023-07-04T18:51:22.311562+0000","last_active":"2023-07-04T18:51:23.417648+0000","last_peered":"2023-07-04T18:51:23.417648+0000","last_clean":"2023-07-04T18:51:23.417648+0000","last_became_active":"2023-07-04T18:51:22.311234+0000","last_became_peered":"2023-07-04T18:51:22.311234+0000","last_unstale":"2023-07-04T18:51:23.417648+0000","last_undegraded":"2023-07-04T18:51:23.417648+0000","last_fullsized":"2023-07-04T18:51:23.417648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.327028+0000","last_change":"2023-07-04T18:51:22.325120+0000","last_active":"2023-07-04T18:51:22.327028+0000","last_peered":"2023-07-04T18:51:22.327028+0000","last_clean":"2023-07-04T18:51:22.327028+0000","last_became_active":"2023-07-04T18:51:22.324834+0000","last_became_peered":"2023-07-04T18:51:22.324834+0000","last_unstale":"2023-07-04T18:51:22.327028+0000","last_undegraded":"2023-07-04T18:51:22.327028+0000","last_fullsized":"2023-07-04T18:51:22.327028+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304437+0000","last_change":"2023-07-04T18:51:21.460823+0000","last_active":"2023-07-04T18:51:23.304437+0000","last_peered":"2023-07-04T18:51:23.304437+0000","last_clean":"2023-07-04T18:51:23.304437+0000","last_became_active":"2023-07-04T18:51:21.460492+0000","last_became_peered":"2023-07-04T18:51:21.460492+0000","last_unstale":"2023-07-04T18:51:23.304437+0000","last_undegraded":"2023-07-04T18:51:23.304437+0000","last_fullsized":"2023-07-04T18:51:23.304437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303755+0000","last_change":"2023-07-04T18:51:22.313093+0000","last_active":"2023-07-04T18:51:23.303755+0000","last_peered":"2023-07-04T18:51:23.303755+0000","last_clean":"2023-07-04T18:51:23.303755+0000","last_became_active":"2023-07-04T18:51:22.312832+0000","last_became_peered":"2023-07-04T18:51:22.312832+0000","last_unstale":"2023-07-04T18:51:23.303755+0000","last_undegraded":"2023-07-04T18:51:23.303755+0000","last_fullsized":"2023-07-04T18:51:23.303755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417409+0000","last_change":"2023-07-04T18:51:21.471442+0000","last_active":"2023-07-04T18:51:23.417409+0000","last_peered":"2023-07-04T18:51:23.417409+0000","last_clean":"2023-07-04T18:51:23.417409+0000","last_became_active":"2023-07-04T18:51:21.471130+0000","last_became_peered":"2023-07-04T18:51:21.471130+0000","last_unstale":"2023-07-04T18:51:23.417409+0000","last_undegraded":"2023-07-04T18:51:23.417409+0000","last_fullsized":"2023-07-04T18:51:23.417409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:38.465858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324053+0000","last_change":"2023-07-04T18:51:22.322483+0000","last_active":"2023-07-04T18:51:22.324053+0000","last_peered":"2023-07-04T18:51:22.324053+0000","last_clean":"2023-07-04T18:51:22.324053+0000","last_became_active":"2023-07-04T18:51:22.322262+0000","last_became_peered":"2023-07-04T18:51:22.322262+0000","last_unstale":"2023-07-04T18:51:22.324053+0000","last_undegraded":"2023-07-04T18:51:22.324053+0000","last_fullsized":"2023-07-04T18:51:22.324053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:28:16.422678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303402+0000","last_change":"2023-07-04T18:51:21.470525+0000","last_active":"2023-07-04T18:51:23.303402+0000","last_peered":"2023-07-04T18:51:23.303402+0000","last_clean":"2023-07-04T18:51:23.303402+0000","last_became_active":"2023-07-04T18:51:21.470230+0000","last_became_peered":"2023-07-04T18:51:21.470230+0000","last_unstale":"2023-07-04T18:51:23.303402+0000","last_undegraded":"2023-07-04T18:51:23.303402+0000","last_fullsized":"2023-07-04T18:51:23.303402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303729+0000","last_change":"2023-07-04T18:51:22.323982+0000","last_active":"2023-07-04T18:51:23.303729+0000","last_peered":"2023-07-04T18:51:23.303729+0000","last_clean":"2023-07-04T18:51:23.303729+0000","last_became_active":"2023-07-04T18:51:22.323650+0000","last_became_peered":"2023-07-04T18:51:22.323650+0000","last_unstale":"2023-07-04T18:51:23.303729+0000","last_undegraded":"2023-07-04T18:51:23.303729+0000","last_fullsized":"2023-07-04T18:51:23.303729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309774+0000","last_change":"2023-07-04T18:51:21.464672+0000","last_active":"2023-07-04T18:51:22.309774+0000","last_peered":"2023-07-04T18:51:22.309774+0000","last_clean":"2023-07-04T18:51:22.309774+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:22.309774+0000","last_undegraded":"2023-07-04T18:51:22.309774+0000","last_fullsized":"2023-07-04T18:51:22.309774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:06.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303369+0000","last_change":"2023-07-04T18:51:22.311834+0000","last_active":"2023-07-04T18:51:23.303369+0000","last_peered":"2023-07-04T18:51:23.303369+0000","last_clean":"2023-07-04T18:51:23.303369+0000","last_became_active":"2023-07-04T18:51:22.311536+0000","last_became_peered":"2023-07-04T18:51:22.311536+0000","last_unstale":"2023-07-04T18:51:23.303369+0000","last_undegraded":"2023-07-04T18:51:23.303369+0000","last_fullsized":"2023-07-04T18:51:23.303369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.319384+0000","last_change":"2023-07-04T18:51:21.464692+0000","last_active":"2023-07-04T18:51:22.319384+0000","last_peered":"2023-07-04T18:51:22.319384+0000","last_clean":"2023-07-04T18:51:22.319384+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:22.319384+0000","last_undegraded":"2023-07-04T18:51:22.319384+0000","last_fullsized":"2023-07-04T18:51:22.319384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303533+0000","last_change":"2023-07-04T18:51:22.320250+0000","last_active":"2023-07-04T18:51:23.303533+0000","last_peered":"2023-07-04T18:51:23.303533+0000","last_clean":"2023-07-04T18:51:23.303533+0000","last_became_active":"2023-07-04T18:51:22.319730+0000","last_became_peered":"2023-07-04T18:51:22.319730+0000","last_unstale":"2023-07-04T18:51:23.303533+0000","last_undegraded":"2023-07-04T18:51:23.303533+0000","last_fullsized":"2023-07-04T18:51:23.303533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451104+0000","last_change":"2023-07-04T18:51:21.471723+0000","last_active":"2023-07-04T18:51:23.451104+0000","last_peered":"2023-07-04T18:51:23.451104+0000","last_clean":"2023-07-04T18:51:23.451104+0000","last_became_active":"2023-07-04T18:51:21.471500+0000","last_became_peered":"2023-07-04T18:51:21.471500+0000","last_unstale":"2023-07-04T18:51:23.451104+0000","last_undegraded":"2023-07-04T18:51:23.451104+0000","last_fullsized":"2023-07-04T18:51:23.451104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:14:51.732204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417449+0000","last_change":"2023-07-04T18:51:22.311576+0000","last_active":"2023-07-04T18:51:23.417449+0000","last_peered":"2023-07-04T18:51:23.417449+0000","last_clean":"2023-07-04T18:51:23.417449+0000","last_became_active":"2023-07-04T18:51:22.311212+0000","last_became_peered":"2023-07-04T18:51:22.311212+0000","last_unstale":"2023-07-04T18:51:23.417449+0000","last_undegraded":"2023-07-04T18:51:23.417449+0000","last_fullsized":"2023-07-04T18:51:23.417449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304622+0000","last_change":"2023-07-04T18:51:21.464597+0000","last_active":"2023-07-04T18:51:23.304622+0000","last_peered":"2023-07-04T18:51:23.304622+0000","last_clean":"2023-07-04T18:51:23.304622+0000","last_became_active":"2023-07-04T18:51:21.464263+0000","last_became_peered":"2023-07-04T18:51:21.464263+0000","last_unstale":"2023-07-04T18:51:23.304622+0000","last_undegraded":"2023-07-04T18:51:23.304622+0000","last_fullsized":"2023-07-04T18:51:23.304622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304535+0000","last_change":"2023-07-04T18:51:22.318474+0000","last_active":"2023-07-04T18:51:23.304535+0000","last_peered":"2023-07-04T18:51:23.304535+0000","last_clean":"2023-07-04T18:51:23.304535+0000","last_became_active":"2023-07-04T18:51:22.317476+0000","last_became_peered":"2023-07-04T18:51:22.317476+0000","last_unstale":"2023-07-04T18:51:23.304535+0000","last_undegraded":"2023-07-04T18:51:23.304535+0000","last_fullsized":"2023-07-04T18:51:23.304535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450969+0000","last_change":"2023-07-04T18:51:21.467028+0000","last_active":"2023-07-04T18:51:23.450969+0000","last_peered":"2023-07-04T18:51:23.450969+0000","last_clean":"2023-07-04T18:51:23.450969+0000","last_became_active":"2023-07-04T18:51:21.466697+0000","last_became_peered":"2023-07-04T18:51:21.466697+0000","last_unstale":"2023-07-04T18:51:23.450969+0000","last_undegraded":"2023-07-04T18:51:23.450969+0000","last_fullsized":"2023-07-04T18:51:23.450969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303917+0000","last_change":"2023-07-04T18:51:22.320412+0000","last_active":"2023-07-04T18:51:23.303917+0000","last_peered":"2023-07-04T18:51:23.303917+0000","last_clean":"2023-07-04T18:51:23.303917+0000","last_became_active":"2023-07-04T18:51:22.320022+0000","last_became_peered":"2023-07-04T18:51:22.320022+0000","last_unstale":"2023-07-04T18:51:23.303917+0000","last_undegraded":"2023-07-04T18:51:23.303917+0000","last_fullsized":"2023-07-04T18:51:23.303917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309984+0000","last_change":"2023-07-04T18:51:21.465044+0000","last_active":"2023-07-04T18:51:22.309984+0000","last_peered":"2023-07-04T18:51:22.309984+0000","last_clean":"2023-07-04T18:51:22.309984+0000","last_became_active":"2023-07-04T18:51:21.464683+0000","last_became_peered":"2023-07-04T18:51:21.464683+0000","last_unstale":"2023-07-04T18:51:22.309984+0000","last_undegraded":"2023-07-04T18:51:22.309984+0000","last_fullsized":"2023-07-04T18:51:22.309984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:57.698962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.326905+0000","last_change":"2023-07-04T18:51:22.324545+0000","last_active":"2023-07-04T18:51:22.326905+0000","last_peered":"2023-07-04T18:51:22.326905+0000","last_clean":"2023-07-04T18:51:22.326905+0000","last_became_active":"2023-07-04T18:51:22.324168+0000","last_became_peered":"2023-07-04T18:51:22.324168+0000","last_unstale":"2023-07-04T18:51:22.326905+0000","last_undegraded":"2023-07-04T18:51:22.326905+0000","last_fullsized":"2023-07-04T18:51:22.326905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:20.924771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450810+0000","last_change":"2023-07-04T18:51:21.468878+0000","last_active":"2023-07-04T18:51:23.450810+0000","last_peered":"2023-07-04T18:51:23.450810+0000","last_clean":"2023-07-04T18:51:23.450810+0000","last_became_active":"2023-07-04T18:51:21.468229+0000","last_became_peered":"2023-07-04T18:51:21.468229+0000","last_unstale":"2023-07-04T18:51:23.450810+0000","last_undegraded":"2023-07-04T18:51:23.450810+0000","last_fullsized":"2023-07-04T18:51:23.450810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320226+0000","last_change":"2023-07-04T18:51:22.317975+0000","last_active":"2023-07-04T18:51:22.320226+0000","last_peered":"2023-07-04T18:51:22.320226+0000","last_clean":"2023-07-04T18:51:22.320226+0000","last_became_active":"2023-07-04T18:51:22.317731+0000","last_became_peered":"2023-07-04T18:51:22.317731+0000","last_unstale":"2023-07-04T18:51:22.320226+0000","last_undegraded":"2023-07-04T18:51:22.320226+0000","last_fullsized":"2023-07-04T18:51:22.320226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303059+0000","last_change":"2023-07-04T18:51:21.460954+0000","last_active":"2023-07-04T18:51:23.303059+0000","last_peered":"2023-07-04T18:51:23.303059+0000","last_clean":"2023-07-04T18:51:23.303059+0000","last_became_active":"2023-07-04T18:51:21.460663+0000","last_became_peered":"2023-07-04T18:51:21.460663+0000","last_unstale":"2023-07-04T18:51:23.303059+0000","last_undegraded":"2023-07-04T18:51:23.303059+0000","last_fullsized":"2023-07-04T18:51:23.303059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:08.443319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304053+0000","last_change":"2023-07-04T18:51:22.313798+0000","last_active":"2023-07-04T18:51:23.304053+0000","last_peered":"2023-07-04T18:51:23.304053+0000","last_clean":"2023-07-04T18:51:23.304053+0000","last_became_active":"2023-07-04T18:51:22.313526+0000","last_became_peered":"2023-07-04T18:51:22.313526+0000","last_unstale":"2023-07-04T18:51:23.304053+0000","last_undegraded":"2023-07-04T18:51:23.304053+0000","last_fullsized":"2023-07-04T18:51:23.304053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451341+0000","last_change":"2023-07-04T18:51:21.468130+0000","last_active":"2023-07-04T18:51:23.451341+0000","last_peered":"2023-07-04T18:51:23.451341+0000","last_clean":"2023-07-04T18:51:23.451341+0000","last_became_active":"2023-07-04T18:51:21.467832+0000","last_became_peered":"2023-07-04T18:51:21.467832+0000","last_unstale":"2023-07-04T18:51:23.451341+0000","last_undegraded":"2023-07-04T18:51:23.451341+0000","last_fullsized":"2023-07-04T18:51:23.451341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:02:58.666739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.327043+0000","last_change":"2023-07-04T18:51:22.325399+0000","last_active":"2023-07-04T18:51:22.327043+0000","last_peered":"2023-07-04T18:51:22.327043+0000","last_clean":"2023-07-04T18:51:22.327043+0000","last_became_active":"2023-07-04T18:51:22.324669+0000","last_became_peered":"2023-07-04T18:51:22.324669+0000","last_unstale":"2023-07-04T18:51:22.327043+0000","last_undegraded":"2023-07-04T18:51:22.327043+0000","last_fullsized":"2023-07-04T18:51:22.327043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:18.636942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451202+0000","last_change":"2023-07-04T18:51:21.466954+0000","last_active":"2023-07-04T18:51:23.451202+0000","last_peered":"2023-07-04T18:51:23.451202+0000","last_clean":"2023-07-04T18:51:23.451202+0000","last_became_active":"2023-07-04T18:51:21.466625+0000","last_became_peered":"2023-07-04T18:51:21.466625+0000","last_unstale":"2023-07-04T18:51:23.451202+0000","last_undegraded":"2023-07-04T18:51:23.451202+0000","last_fullsized":"2023-07-04T18:51:23.451202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:17:35.188150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304676+0000","last_change":"2023-07-04T18:51:22.321314+0000","last_active":"2023-07-04T18:51:23.304676+0000","last_peered":"2023-07-04T18:51:23.304676+0000","last_clean":"2023-07-04T18:51:23.304676+0000","last_became_active":"2023-07-04T18:51:22.321008+0000","last_became_peered":"2023-07-04T18:51:22.321008+0000","last_unstale":"2023-07-04T18:51:23.304676+0000","last_undegraded":"2023-07-04T18:51:23.304676+0000","last_fullsized":"2023-07-04T18:51:23.304676+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451276+0000","last_change":"2023-07-04T18:51:21.468101+0000","last_active":"2023-07-04T18:51:23.451276+0000","last_peered":"2023-07-04T18:51:23.451276+0000","last_clean":"2023-07-04T18:51:23.451276+0000","last_became_active":"2023-07-04T18:51:21.467813+0000","last_became_peered":"2023-07-04T18:51:21.467813+0000","last_unstale":"2023-07-04T18:51:23.451276+0000","last_undegraded":"2023-07-04T18:51:23.451276+0000","last_fullsized":"2023-07-04T18:51:23.451276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:02:10.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303594+0000","last_change":"2023-07-04T18:51:22.324752+0000","last_active":"2023-07-04T18:51:23.303594+0000","last_peered":"2023-07-04T18:51:23.303594+0000","last_clean":"2023-07-04T18:51:23.303594+0000","last_became_active":"2023-07-04T18:51:22.324420+0000","last_became_peered":"2023-07-04T18:51:22.324420+0000","last_unstale":"2023-07-04T18:51:23.303594+0000","last_undegraded":"2023-07-04T18:51:23.303594+0000","last_fullsized":"2023-07-04T18:51:23.303594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:06.453079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316725+0000","last_change":"2023-07-04T18:51:21.471602+0000","last_active":"2023-07-04T18:51:22.316725+0000","last_peered":"2023-07-04T18:51:22.316725+0000","last_clean":"2023-07-04T18:51:22.316725+0000","last_became_active":"2023-07-04T18:51:21.471319+0000","last_became_peered":"2023-07-04T18:51:21.471319+0000","last_unstale":"2023-07-04T18:51:22.316725+0000","last_undegraded":"2023-07-04T18:51:22.316725+0000","last_fullsized":"2023-07-04T18:51:22.316725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:29.652241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.323916+0000","last_change":"2023-07-04T18:51:22.321087+0000","last_active":"2023-07-04T18:51:22.323916+0000","last_peered":"2023-07-04T18:51:22.323916+0000","last_clean":"2023-07-04T18:51:22.323916+0000","last_became_active":"2023-07-04T18:51:22.320871+0000","last_became_peered":"2023-07-04T18:51:22.320871+0000","last_unstale":"2023-07-04T18:51:22.323916+0000","last_undegraded":"2023-07-04T18:51:22.323916+0000","last_fullsized":"2023-07-04T18:51:22.323916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:27.753990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.473477+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.473173+0000","last_became_peered":"2023-07-04T18:51:21.473173+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304599+0000","last_change":"2023-07-04T18:51:22.317634+0000","last_active":"2023-07-04T18:51:23.304599+0000","last_peered":"2023-07-04T18:51:23.304599+0000","last_clean":"2023-07-04T18:51:23.304599+0000","last_became_active":"2023-07-04T18:51:22.317260+0000","last_became_peered":"2023-07-04T18:51:22.317260+0000","last_unstale":"2023-07-04T18:51:23.304599+0000","last_undegraded":"2023-07-04T18:51:23.304599+0000","last_fullsized":"2023-07-04T18:51:23.304599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303472+0000","last_change":"2023-07-04T18:51:21.463128+0000","last_active":"2023-07-04T18:51:23.303472+0000","last_peered":"2023-07-04T18:51:23.303472+0000","last_clean":"2023-07-04T18:51:23.303472+0000","last_became_active":"2023-07-04T18:51:21.462860+0000","last_became_peered":"2023-07-04T18:51:21.462860+0000","last_unstale":"2023-07-04T18:51:23.303472+0000","last_undegraded":"2023-07-04T18:51:23.303472+0000","last_fullsized":"2023-07-04T18:51:23.303472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304928+0000","last_change":"2023-07-04T18:51:22.318985+0000","last_active":"2023-07-04T18:51:23.304928+0000","last_peered":"2023-07-04T18:51:23.304928+0000","last_clean":"2023-07-04T18:51:23.304928+0000","last_became_active":"2023-07-04T18:51:22.318393+0000","last_became_peered":"2023-07-04T18:51:22.318393+0000","last_unstale":"2023-07-04T18:51:23.304928+0000","last_undegraded":"2023-07-04T18:51:23.304928+0000","last_fullsized":"2023-07-04T18:51:23.304928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304517+0000","last_change":"2023-07-04T18:51:21.466769+0000","last_active":"2023-07-04T18:51:23.304517+0000","last_peered":"2023-07-04T18:51:23.304517+0000","last_clean":"2023-07-04T18:51:23.304517+0000","last_became_active":"2023-07-04T18:51:21.466504+0000","last_became_peered":"2023-07-04T18:51:21.466504+0000","last_unstale":"2023-07-04T18:51:23.304517+0000","last_undegraded":"2023-07-04T18:51:23.304517+0000","last_fullsized":"2023-07-04T18:51:23.304517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:03.327143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304757+0000","last_change":"2023-07-04T18:51:22.317855+0000","last_active":"2023-07-04T18:51:23.304757+0000","last_peered":"2023-07-04T18:51:23.304757+0000","last_clean":"2023-07-04T18:51:23.304757+0000","last_became_active":"2023-07-04T18:51:22.317426+0000","last_became_peered":"2023-07-04T18:51:22.317426+0000","last_unstale":"2023-07-04T18:51:23.304757+0000","last_undegraded":"2023-07-04T18:51:23.304757+0000","last_fullsized":"2023-07-04T18:51:23.304757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303314+0000","last_change":"2023-07-04T18:51:21.471306+0000","last_active":"2023-07-04T18:51:23.303314+0000","last_peered":"2023-07-04T18:51:23.303314+0000","last_clean":"2023-07-04T18:51:23.303314+0000","last_became_active":"2023-07-04T18:51:21.471034+0000","last_became_peered":"2023-07-04T18:51:21.471034+0000","last_unstale":"2023-07-04T18:51:23.303314+0000","last_undegraded":"2023-07-04T18:51:23.303314+0000","last_fullsized":"2023-07-04T18:51:23.303314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417826+0000","last_change":"2023-07-04T18:51:22.315022+0000","last_active":"2023-07-04T18:51:23.417826+0000","last_peered":"2023-07-04T18:51:23.417826+0000","last_clean":"2023-07-04T18:51:23.417826+0000","last_became_active":"2023-07-04T18:51:22.314767+0000","last_became_peered":"2023-07-04T18:51:22.314767+0000","last_unstale":"2023-07-04T18:51:23.417826+0000","last_undegraded":"2023-07-04T18:51:23.417826+0000","last_fullsized":"2023-07-04T18:51:23.417826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417511+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:51:23.417511+0000","last_peered":"2023-07-04T18:51:23.417511+0000","last_clean":"2023-07-04T18:51:23.417511+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:51:23.417511+0000","last_undegraded":"2023-07-04T18:51:23.417511+0000","last_fullsized":"2023-07-04T18:51:23.417511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.317007+0000","last_change":"2023-07-04T18:51:21.471879+0000","last_active":"2023-07-04T18:51:22.317007+0000","last_peered":"2023-07-04T18:51:22.317007+0000","last_clean":"2023-07-04T18:51:22.317007+0000","last_became_active":"2023-07-04T18:51:21.471549+0000","last_became_peered":"2023-07-04T18:51:21.471549+0000","last_unstale":"2023-07-04T18:51:22.317007+0000","last_undegraded":"2023-07-04T18:51:22.317007+0000","last_fullsized":"2023-07-04T18:51:22.317007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:50.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417554+0000","last_change":"2023-07-04T18:51:22.319180+0000","last_active":"2023-07-04T18:51:23.417554+0000","last_peered":"2023-07-04T18:51:23.417554+0000","last_clean":"2023-07-04T18:51:23.417554+0000","last_became_active":"2023-07-04T18:51:22.318906+0000","last_became_peered":"2023-07-04T18:51:22.318906+0000","last_unstale":"2023-07-04T18:51:23.417554+0000","last_undegraded":"2023-07-04T18:51:23.417554+0000","last_fullsized":"2023-07-04T18:51:23.417554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316574+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:51:22.316574+0000","last_peered":"2023-07-04T18:51:22.316574+0000","last_clean":"2023-07-04T18:51:22.316574+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:51:22.316574+0000","last_undegraded":"2023-07-04T18:51:22.316574+0000","last_fullsized":"2023-07-04T18:51:22.316574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:01.315083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304959+0000","last_change":"2023-07-04T18:51:21.461532+0000","last_active":"2023-07-04T18:51:23.304959+0000","last_peered":"2023-07-04T18:51:23.304959+0000","last_clean":"2023-07-04T18:51:23.304959+0000","last_became_active":"2023-07-04T18:51:21.460816+0000","last_became_peered":"2023-07-04T18:51:21.460816+0000","last_unstale":"2023-07-04T18:51:23.304959+0000","last_undegraded":"2023-07-04T18:51:23.304959+0000","last_fullsized":"2023-07-04T18:51:23.304959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417715+0000","last_change":"2023-07-04T18:51:22.312480+0000","last_active":"2023-07-04T18:51:23.417715+0000","last_peered":"2023-07-04T18:51:23.417715+0000","last_clean":"2023-07-04T18:51:23.417715+0000","last_became_active":"2023-07-04T18:51:22.311193+0000","last_became_peered":"2023-07-04T18:51:22.311193+0000","last_unstale":"2023-07-04T18:51:23.417715+0000","last_undegraded":"2023-07-04T18:51:23.417715+0000","last_fullsized":"2023-07-04T18:51:23.417715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.301750+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:51:22.301750+0000","last_peered":"2023-07-04T18:51:22.301750+0000","last_clean":"2023-07-04T18:51:22.301750+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:51:22.301750+0000","last_undegraded":"2023-07-04T18:51:22.301750+0000","last_fullsized":"2023-07-04T18:51:22.301750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:36:47.185435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417530+0000","last_change":"2023-07-04T18:51:21.467449+0000","last_active":"2023-07-04T18:51:23.417530+0000","last_peered":"2023-07-04T18:51:23.417530+0000","last_clean":"2023-07-04T18:51:23.417530+0000","last_became_active":"2023-07-04T18:51:21.466522+0000","last_became_peered":"2023-07-04T18:51:21.466522+0000","last_unstale":"2023-07-04T18:51:23.417530+0000","last_undegraded":"2023-07-04T18:51:23.417530+0000","last_fullsized":"2023-07-04T18:51:23.417530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417692+0000","last_change":"2023-07-04T18:51:22.324258+0000","last_active":"2023-07-04T18:51:23.417692+0000","last_peered":"2023-07-04T18:51:23.417692+0000","last_clean":"2023-07-04T18:51:23.417692+0000","last_became_active":"2023-07-04T18:51:22.323966+0000","last_became_peered":"2023-07-04T18:51:22.323966+0000","last_unstale":"2023-07-04T18:51:23.417692+0000","last_undegraded":"2023-07-04T18:51:23.417692+0000","last_fullsized":"2023-07-04T18:51:23.417692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316509+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:51:22.316509+0000","last_peered":"2023-07-04T18:51:22.316509+0000","last_clean":"2023-07-04T18:51:22.316509+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:51:22.316509+0000","last_undegraded":"2023-07-04T18:51:22.316509+0000","last_fullsized":"2023-07-04T18:51:22.316509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450698+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:51:23.450698+0000","last_peered":"2023-07-04T18:51:23.450698+0000","last_clean":"2023-07-04T18:51:23.450698+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:51:23.450698+0000","last_undegraded":"2023-07-04T18:51:23.450698+0000","last_fullsized":"2023-07-04T18:51:23.450698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:37.197069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303756+0000","last_change":"2023-07-04T18:51:21.470626+0000","last_active":"2023-07-04T18:51:23.303756+0000","last_peered":"2023-07-04T18:51:23.303756+0000","last_clean":"2023-07-04T18:51:23.303756+0000","last_became_active":"2023-07-04T18:51:21.470351+0000","last_became_peered":"2023-07-04T18:51:21.470351+0000","last_unstale":"2023-07-04T18:51:23.303756+0000","last_undegraded":"2023-07-04T18:51:23.303756+0000","last_fullsized":"2023-07-04T18:51:23.303756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303308+0000","last_change":"2023-07-04T18:51:22.321449+0000","last_active":"2023-07-04T18:51:23.303308+0000","last_peered":"2023-07-04T18:51:23.303308+0000","last_clean":"2023-07-04T18:51:23.303308+0000","last_became_active":"2023-07-04T18:51:22.321158+0000","last_became_peered":"2023-07-04T18:51:22.321158+0000","last_unstale":"2023-07-04T18:51:23.303308+0000","last_undegraded":"2023-07-04T18:51:23.303308+0000","last_fullsized":"2023-07-04T18:51:23.303308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450897+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:51:23.450897+0000","last_peered":"2023-07-04T18:51:23.450897+0000","last_clean":"2023-07-04T18:51:23.450897+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:51:23.450897+0000","last_undegraded":"2023-07-04T18:51:23.450897+0000","last_fullsized":"2023-07-04T18:51:23.450897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:48.113610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309128+0000","last_change":"2023-07-04T18:51:21.460314+0000","last_active":"2023-07-04T18:51:22.309128+0000","last_peered":"2023-07-04T18:51:22.309128+0000","last_clean":"2023-07-04T18:51:22.309128+0000","last_became_active":"2023-07-04T18:51:21.460022+0000","last_became_peered":"2023-07-04T18:51:21.460022+0000","last_unstale":"2023-07-04T18:51:22.309128+0000","last_undegraded":"2023-07-04T18:51:22.309128+0000","last_fullsized":"2023-07-04T18:51:22.309128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:17.635315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324037+0000","last_change":"2023-07-04T18:51:22.321571+0000","last_active":"2023-07-04T18:51:22.324037+0000","last_peered":"2023-07-04T18:51:22.324037+0000","last_clean":"2023-07-04T18:51:22.324037+0000","last_became_active":"2023-07-04T18:51:22.321326+0000","last_became_peered":"2023-07-04T18:51:22.321326+0000","last_unstale":"2023-07-04T18:51:22.324037+0000","last_undegraded":"2023-07-04T18:51:22.324037+0000","last_fullsized":"2023-07-04T18:51:22.324037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:20.924771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451151+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:51:23.451151+0000","last_peered":"2023-07-04T18:51:23.451151+0000","last_clean":"2023-07-04T18:51:23.451151+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:51:23.451151+0000","last_undegraded":"2023-07-04T18:51:23.451151+0000","last_fullsized":"2023-07-04T18:51:23.451151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:14:08.639700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417718+0000","last_change":"2023-07-04T18:51:21.466845+0000","last_active":"2023-07-04T18:51:23.417718+0000","last_peered":"2023-07-04T18:51:23.417718+0000","last_clean":"2023-07-04T18:51:23.417718+0000","last_became_active":"2023-07-04T18:51:21.466546+0000","last_became_peered":"2023-07-04T18:51:21.466546+0000","last_unstale":"2023-07-04T18:51:23.417718+0000","last_undegraded":"2023-07-04T18:51:23.417718+0000","last_fullsized":"2023-07-04T18:51:23.417718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.321659+0000","last_change":"2023-07-04T18:51:22.318511+0000","last_active":"2023-07-04T18:51:22.321659+0000","last_peered":"2023-07-04T18:51:22.321659+0000","last_clean":"2023-07-04T18:51:22.321659+0000","last_became_active":"2023-07-04T18:51:22.318227+0000","last_became_peered":"2023-07-04T18:51:22.318227+0000","last_unstale":"2023-07-04T18:51:22.321659+0000","last_undegraded":"2023-07-04T18:51:22.321659+0000","last_fullsized":"2023-07-04T18:51:22.321659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:31:55.772208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304031+0000","last_change":"2023-07-04T18:51:21.474661+0000","last_active":"2023-07-04T18:51:23.304031+0000","last_peered":"2023-07-04T18:51:23.304031+0000","last_clean":"2023-07-04T18:51:23.304031+0000","last_became_active":"2023-07-04T18:51:21.474400+0000","last_became_peered":"2023-07-04T18:51:21.474400+0000","last_unstale":"2023-07-04T18:51:23.304031+0000","last_undegraded":"2023-07-04T18:51:23.304031+0000","last_fullsized":"2023-07-04T18:51:23.304031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417697+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:51:23.417697+0000","last_peered":"2023-07-04T18:51:23.417697+0000","last_clean":"2023-07-04T18:51:23.417697+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:51:23.417697+0000","last_undegraded":"2023-07-04T18:51:23.417697+0000","last_fullsized":"2023-07-04T18:51:23.417697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:02.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303243+0000","last_change":"2023-07-04T18:51:22.313754+0000","last_active":"2023-07-04T18:51:23.303243+0000","last_peered":"2023-07-04T18:51:23.303243+0000","last_clean":"2023-07-04T18:51:23.303243+0000","last_became_active":"2023-07-04T18:51:22.313509+0000","last_became_peered":"2023-07-04T18:51:22.313509+0000","last_unstale":"2023-07-04T18:51:23.303243+0000","last_undegraded":"2023-07-04T18:51:23.303243+0000","last_fullsized":"2023-07-04T18:51:23.303243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303419+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:51:23.303419+0000","last_peered":"2023-07-04T18:51:23.303419+0000","last_clean":"2023-07-04T18:51:23.303419+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:51:23.303419+0000","last_undegraded":"2023-07-04T18:51:23.303419+0000","last_fullsized":"2023-07-04T18:51:23.303419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309923+0000","last_change":"2023-07-04T18:51:21.464739+0000","last_active":"2023-07-04T18:51:22.309923+0000","last_peered":"2023-07-04T18:51:22.309923+0000","last_clean":"2023-07-04T18:51:22.309923+0000","last_became_active":"2023-07-04T18:51:21.464445+0000","last_became_peered":"2023-07-04T18:51:21.464445+0000","last_unstale":"2023-07-04T18:51:22.309923+0000","last_undegraded":"2023-07-04T18:51:22.309923+0000","last_fullsized":"2023-07-04T18:51:22.309923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:43.919530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304372+0000","last_change":"2023-07-04T18:51:22.326679+0000","last_active":"2023-07-04T18:51:23.304372+0000","last_peered":"2023-07-04T18:51:23.304372+0000","last_clean":"2023-07-04T18:51:23.304372+0000","last_became_active":"2023-07-04T18:51:22.326348+0000","last_became_peered":"2023-07-04T18:51:22.326348+0000","last_unstale":"2023-07-04T18:51:23.304372+0000","last_undegraded":"2023-07-04T18:51:23.304372+0000","last_fullsized":"2023-07-04T18:51:23.304372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.486397+0000","last_change":"2023-07-04T18:51:21.466007+0000","last_active":"2023-07-04T18:51:22.486397+0000","last_peered":"2023-07-04T18:51:22.486397+0000","last_clean":"2023-07-04T18:51:22.486397+0000","last_became_active":"2023-07-04T18:51:21.465754+0000","last_became_peered":"2023-07-04T18:51:21.465754+0000","last_unstale":"2023-07-04T18:51:22.486397+0000","last_undegraded":"2023-07-04T18:51:22.486397+0000","last_fullsized":"2023-07-04T18:51:22.486397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:41.484807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315223+0000","last_change":"2023-07-04T18:51:22.309513+0000","last_active":"2023-07-04T18:51:22.315223+0000","last_peered":"2023-07-04T18:51:22.315223+0000","last_clean":"2023-07-04T18:51:22.315223+0000","last_became_active":"2023-07-04T18:51:22.309252+0000","last_became_peered":"2023-07-04T18:51:22.309252+0000","last_unstale":"2023-07-04T18:51:22.315223+0000","last_undegraded":"2023-07-04T18:51:22.315223+0000","last_fullsized":"2023-07-04T18:51:22.315223+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:34:07.018645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.486707+0000","last_change":"2023-07-04T18:51:21.463261+0000","last_active":"2023-07-04T18:51:22.486707+0000","last_peered":"2023-07-04T18:51:22.486707+0000","last_clean":"2023-07-04T18:51:22.486707+0000","last_became_active":"2023-07-04T18:51:21.462985+0000","last_became_peered":"2023-07-04T18:51:21.462985+0000","last_unstale":"2023-07-04T18:51:22.486707+0000","last_undegraded":"2023-07-04T18:51:22.486707+0000","last_fullsized":"2023-07-04T18:51:22.486707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304859+0000","last_change":"2023-07-04T18:51:22.318260+0000","last_active":"2023-07-04T18:51:23.304859+0000","last_peered":"2023-07-04T18:51:23.304859+0000","last_clean":"2023-07-04T18:51:23.304859+0000","last_became_active":"2023-07-04T18:51:22.317037+0000","last_became_peered":"2023-07-04T18:51:22.317037+0000","last_unstale":"2023-07-04T18:51:23.304859+0000","last_undegraded":"2023-07-04T18:51:23.304859+0000","last_fullsized":"2023-07-04T18:51:23.304859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.487409+0000","last_change":"2023-07-04T18:51:21.463234+0000","last_active":"2023-07-04T18:51:22.487409+0000","last_peered":"2023-07-04T18:51:22.487409+0000","last_clean":"2023-07-04T18:51:22.487409+0000","last_became_active":"2023-07-04T18:51:21.462954+0000","last_became_peered":"2023-07-04T18:51:21.462954+0000","last_unstale":"2023-07-04T18:51:22.487409+0000","last_undegraded":"2023-07-04T18:51:22.487409+0000","last_fullsized":"2023-07-04T18:51:22.487409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:09.767759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304612+0000","last_change":"2023-07-04T18:51:22.317628+0000","last_active":"2023-07-04T18:51:23.304612+0000","last_peered":"2023-07-04T18:51:23.304612+0000","last_clean":"2023-07-04T18:51:23.304612+0000","last_became_active":"2023-07-04T18:51:22.317043+0000","last_became_peered":"2023-07-04T18:51:22.317043+0000","last_unstale":"2023-07-04T18:51:23.304612+0000","last_undegraded":"2023-07-04T18:51:23.304612+0000","last_fullsized":"2023-07-04T18:51:23.304612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.470557+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.470247+0000","last_became_peered":"2023-07-04T18:51:21.470247+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:13.171949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303469+0000","last_change":"2023-07-04T18:51:22.313146+0000","last_active":"2023-07-04T18:51:23.303469+0000","last_peered":"2023-07-04T18:51:23.303469+0000","last_clean":"2023-07-04T18:51:23.303469+0000","last_became_active":"2023-07-04T18:51:22.312877+0000","last_became_peered":"2023-07-04T18:51:22.312877+0000","last_unstale":"2023-07-04T18:51:23.303469+0000","last_undegraded":"2023-07-04T18:51:23.303469+0000","last_fullsized":"2023-07-04T18:51:23.303469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.489888+0000","last_change":"2023-07-04T18:51:21.459772+0000","last_active":"2023-07-04T18:51:22.489888+0000","last_peered":"2023-07-04T18:51:22.489888+0000","last_clean":"2023-07-04T18:51:22.489888+0000","last_became_active":"2023-07-04T18:51:21.459459+0000","last_became_peered":"2023-07-04T18:51:21.459459+0000","last_unstale":"2023-07-04T18:51:22.489888+0000","last_undegraded":"2023-07-04T18:51:22.489888+0000","last_fullsized":"2023-07-04T18:51:22.489888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303922+0000","last_change":"2023-07-04T18:51:22.318229+0000","last_active":"2023-07-04T18:51:23.303922+0000","last_peered":"2023-07-04T18:51:23.303922+0000","last_clean":"2023-07-04T18:51:23.303922+0000","last_became_active":"2023-07-04T18:51:22.317388+0000","last_became_peered":"2023-07-04T18:51:22.317388+0000","last_unstale":"2023-07-04T18:51:23.303922+0000","last_undegraded":"2023-07-04T18:51:23.303922+0000","last_fullsized":"2023-07-04T18:51:23.303922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:55.968814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303155+0000","last_change":"2023-07-04T18:51:22.314933+0000","last_active":"2023-07-04T18:51:23.303155+0000","last_peered":"2023-07-04T18:51:23.303155+0000","last_clean":"2023-07-04T18:51:23.303155+0000","last_became_active":"2023-07-04T18:51:22.314648+0000","last_became_peered":"2023-07-04T18:51:22.314648+0000","last_unstale":"2023-07-04T18:51:23.303155+0000","last_undegraded":"2023-07-04T18:51:23.303155+0000","last_fullsized":"2023-07-04T18:51:23.303155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303365+0000","last_change":"2023-07-04T18:51:22.318422+0000","last_active":"2023-07-04T18:51:23.303365+0000","last_peered":"2023-07-04T18:51:23.303365+0000","last_clean":"2023-07-04T18:51:23.303365+0000","last_became_active":"2023-07-04T18:51:22.318161+0000","last_became_peered":"2023-07-04T18:51:22.318161+0000","last_unstale":"2023-07-04T18:51:23.303365+0000","last_undegraded":"2023-07-04T18:51:23.303365+0000","last_fullsized":"2023-07-04T18:51:23.303365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304314+0000","last_change":"2023-07-04T18:51:22.320144+0000","last_active":"2023-07-04T18:51:23.304314+0000","last_peered":"2023-07-04T18:51:23.304314+0000","last_clean":"2023-07-04T18:51:23.304314+0000","last_became_active":"2023-07-04T18:51:22.317892+0000","last_became_peered":"2023-07-04T18:51:22.317892+0000","last_unstale":"2023-07-04T18:51:23.304314+0000","last_undegraded":"2023-07-04T18:51:23.304314+0000","last_fullsized":"2023-07-04T18:51:23.304314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303934+0000","last_change":"2023-07-04T18:51:22.317154+0000","last_active":"2023-07-04T18:51:23.303934+0000","last_peered":"2023-07-04T18:51:23.303934+0000","last_clean":"2023-07-04T18:51:23.303934+0000","last_became_active":"2023-07-04T18:51:22.316916+0000","last_became_peered":"2023-07-04T18:51:22.316916+0000","last_unstale":"2023-07-04T18:51:23.303934+0000","last_undegraded":"2023-07-04T18:51:23.303934+0000","last_fullsized":"2023-07-04T18:51:23.303934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417430+0000","last_change":"2023-07-04T18:51:22.312307+0000","last_active":"2023-07-04T18:51:23.417430+0000","last_peered":"2023-07-04T18:51:23.417430+0000","last_clean":"2023-07-04T18:51:23.417430+0000","last_became_active":"2023-07-04T18:51:22.310832+0000","last_became_peered":"2023-07-04T18:51:22.310832+0000","last_unstale":"2023-07-04T18:51:23.417430+0000","last_undegraded":"2023-07-04T18:51:23.417430+0000","last_fullsized":"2023-07-04T18:51:23.417430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":864256,"data_stored":681747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":673896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":262144,"data_stored":91219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":237568,"data_stored":83598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:51:26.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:51:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 Processor -- start 2023-07-04T18:51:26.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 -- start start 2023-07-04T18:51:26.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fded815a430 0x7fded80ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fded80cebf0 0x7fded80cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80cf100 con 0x7fded815ea40 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded815df60 con 0x7fded80cebf0 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fdede40b700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded815e240 con 0x7fded815a430 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.729+0000 7fded77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fded80cebf0 0x7fded80cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fded80cebf0 0x7fded80cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43924/0 (socket says 172.21.15.77:43924) 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 -- 172.21.15.77:0/1901722991 learned_addr learned my addr 172.21.15.77:0/1901722991 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 -- 172.21.15.77:0/1901722991 >> v2:172.21.15.82:3300/0 conn(0x7fded815a430 msgr2=0x7fded80ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 --2- 172.21.15.77:0/1901722991 >> v2:172.21.15.82:3300/0 conn(0x7fded815a430 0x7fded80ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 -- 172.21.15.77:0/1901722991 >> v1:172.21.15.82:6789/0 conn(0x7fded815ea40 legacy=0x7fded80cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 -- 172.21.15.77:0/1901722991 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded815e520 con 0x7fded80cebf0 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded77fe700 1 --2- 172.21.15.77:0/1901722991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded80cf3e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fded80d0a40 tx=0x7fdecc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8a0a18f71631a server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded6ffd700 1 -- 172.21.15.77:0/1901722991 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc004230 con 0x7fded80cebf0 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded6ffd700 1 -- 172.21.15.77:0/1901722991 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc004390 con 0x7fded80cebf0 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fded6ffd700 1 -- 172.21.15.77:0/1901722991 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0036a0 con 0x7fded80cebf0 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fdede40b700 1 -- 172.21.15.77:0/1901722991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 msgr2=0x7fded80cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fdede40b700 1 --2- 172.21.15.77:0/1901722991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded80cf3e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fded80d0a40 tx=0x7fdecc009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fdede40b700 1 -- 172.21.15.77:0/1901722991 shutdown_connections 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fdede40b700 1 --2- 172.21.15.77:0/1901722991 >> v2:172.21.15.82:3300/0 conn(0x7fded815a430 0x7fded80ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.730+0000 7fdede40b700 1 --2- 172.21.15.77:0/1901722991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded80cf3e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.731+0000 7fdede40b700 1 -- 172.21.15.77:0/1901722991 >> 172.21.15.77:0/1901722991 conn(0x7fded806d7c0 msgr2=0x7fded806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.731+0000 7fdede40b700 1 -- 172.21.15.77:0/1901722991 shutdown_connections 2023-07-04T18:51:26.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.731+0000 7fdede40b700 1 -- 172.21.15.77:0/1901722991 wait complete. 2023-07-04T18:51:26.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.732+0000 7fdede40b700 1 Processor -- start 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.732+0000 7fdede40b700 1 -- start start 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fded815ea40 0x7fded81515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 0x7fded80cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded8152490 con 0x7fded80cebf0 2023-07-04T18:51:26.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded806c820 con 0x7fded815ea40 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded806ca60 con 0x7fded806c380 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43928/0 (socket says 172.21.15.77:43928) 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 -- 172.21.15.77:0/2305197660 learned_addr learned my addr 172.21.15.77:0/2305197660 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded77fe700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fded815ea40 0x7fded81515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdedc9a8700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 0x7fded80cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 -- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 msgr2=0x7fded80cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 0x7fded80cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 -- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fded815ea40 msgr2=0x7fded81515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fded815ea40 0x7fded81515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 -- 172.21.15.77:0/2305197660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc009550 con 0x7fded80cebf0 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdedc9a8700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 0x7fded80cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:26.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded7fff700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdec800b530 tx=0x7fdec800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8011830 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8011990 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded806cd40 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8011d20 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.733+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded80acd10 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.734+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded80acf00 con 0x7fded80cebf0 2023-07-04T18:51:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.735+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdec801a400 con 0x7fded80cebf0 2023-07-04T18:51:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.735+0000 7fded4ff9700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 0x7fdec406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.735+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdec8089520 con 0x7fded80cebf0 2023-07-04T18:51:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.735+0000 7fded77fe700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 0x7fdec406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:26.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.736+0000 7fded77fe700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 0x7fdec406d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009130 tx=0x7fdecc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:26.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.739+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdec8054660 con 0x7fded80cebf0 2023-07-04T18:51:26.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.922+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fded80ac8b0 con 0x7fdec406ace0 2023-07-04T18:51:26.930 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.930+0000 7fded4ff9700 1 -- 172.21.15.77:0/2305197660 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286694 (secure 0 0 0) 0x7fdecc0041e0 con 0x7fdec406ace0 2023-07-04T18:51:26.933 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 msgr2=0x7fdec406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 0x7fdec406d190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009130 tx=0x7fdecc008030 comp rx=0 tx=0).stop 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 msgr2=0x7fded8151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdec800b530 tx=0x7fdec800b840 comp rx=0 tx=0).stop 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 shutdown_connections 2023-07-04T18:51:26.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdec406ace0 0x7fdec406d190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fded80cebf0 0x7fded8151bf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fded806c380 0x7fded80cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 --2- 172.21.15.77:0/2305197660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fded815ea40 0x7fded81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 >> 172.21.15.77:0/2305197660 conn(0x7fded806d7c0 msgr2=0x7fded80af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 shutdown_connections 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:26.933+0000 7fdede40b700 1 -- 172.21.15.77:0/2305197660 wait complete. 2023-07-04T18:51:26.937 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:51:26.945 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-07-04T18:51:25.176720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218668,"kb_used_data":3052,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754756052,"statfs":{"total":773094113280,"available":772870197248,"internally_reserved":0,"allocated":3125248,"data_stored":1868987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.733296"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316037+0000","last_change":"2023-07-04T18:51:22.314047+0000","last_active":"2023-07-04T18:51:22.316037+0000","last_peered":"2023-07-04T18:51:22.316037+0000","last_clean":"2023-07-04T18:51:22.316037+0000","last_became_active":"2023-07-04T18:51:22.313772+0000","last_became_peered":"2023-07-04T18:51:22.313772+0000","last_unstale":"2023-07-04T18:51:22.316037+0000","last_undegraded":"2023-07-04T18:51:22.316037+0000","last_fullsized":"2023-07-04T18:51:22.316037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:07:21.587682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325987+0000","last_change":"2023-07-04T18:51:22.324407+0000","last_active":"2023-07-04T18:51:22.325987+0000","last_peered":"2023-07-04T18:51:22.325987+0000","last_clean":"2023-07-04T18:51:22.325987+0000","last_became_active":"2023-07-04T18:51:22.323862+0000","last_became_peered":"2023-07-04T18:51:22.323862+0000","last_unstale":"2023-07-04T18:51:22.325987+0000","last_undegraded":"2023-07-04T18:51:22.325987+0000","last_fullsized":"2023-07-04T18:51:22.325987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:46.737923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304530+0000","last_change":"2023-07-04T18:51:22.319081+0000","last_active":"2023-07-04T18:51:23.304530+0000","last_peered":"2023-07-04T18:51:23.304530+0000","last_clean":"2023-07-04T18:51:23.304530+0000","last_became_active":"2023-07-04T18:51:22.318806+0000","last_became_peered":"2023-07-04T18:51:22.318806+0000","last_unstale":"2023-07-04T18:51:23.304530+0000","last_undegraded":"2023-07-04T18:51:23.304530+0000","last_fullsized":"2023-07-04T18:51:23.304530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303285+0000","last_change":"2023-07-04T18:51:22.319728+0000","last_active":"2023-07-04T18:51:23.303285+0000","last_peered":"2023-07-04T18:51:23.303285+0000","last_clean":"2023-07-04T18:51:23.303285+0000","last_became_active":"2023-07-04T18:51:22.319417+0000","last_became_peered":"2023-07-04T18:51:22.319417+0000","last_unstale":"2023-07-04T18:51:23.303285+0000","last_undegraded":"2023-07-04T18:51:23.303285+0000","last_fullsized":"2023-07-04T18:51:23.303285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320125+0000","last_change":"2023-07-04T18:51:22.317945+0000","last_active":"2023-07-04T18:51:22.320125+0000","last_peered":"2023-07-04T18:51:22.320125+0000","last_clean":"2023-07-04T18:51:22.320125+0000","last_became_active":"2023-07-04T18:51:22.317635+0000","last_became_peered":"2023-07-04T18:51:22.317635+0000","last_unstale":"2023-07-04T18:51:22.320125+0000","last_undegraded":"2023-07-04T18:51:22.320125+0000","last_fullsized":"2023-07-04T18:51:22.320125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:19:18.779329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304259+0000","last_change":"2023-07-04T18:51:22.320918+0000","last_active":"2023-07-04T18:51:23.304259+0000","last_peered":"2023-07-04T18:51:23.304259+0000","last_clean":"2023-07-04T18:51:23.304259+0000","last_became_active":"2023-07-04T18:51:22.320502+0000","last_became_peered":"2023-07-04T18:51:22.320502+0000","last_unstale":"2023-07-04T18:51:23.304259+0000","last_undegraded":"2023-07-04T18:51:23.304259+0000","last_fullsized":"2023-07-04T18:51:23.304259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450875+0000","last_change":"2023-07-04T18:51:22.323607+0000","last_active":"2023-07-04T18:51:23.450875+0000","last_peered":"2023-07-04T18:51:23.450875+0000","last_clean":"2023-07-04T18:51:23.450875+0000","last_became_active":"2023-07-04T18:51:22.322940+0000","last_became_peered":"2023-07-04T18:51:22.322940+0000","last_unstale":"2023-07-04T18:51:23.450875+0000","last_undegraded":"2023-07-04T18:51:23.450875+0000","last_fullsized":"2023-07-04T18:51:23.450875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325268+0000","last_change":"2023-07-04T18:51:22.321665+0000","last_active":"2023-07-04T18:51:22.325268+0000","last_peered":"2023-07-04T18:51:22.325268+0000","last_clean":"2023-07-04T18:51:22.325268+0000","last_became_active":"2023-07-04T18:51:22.321426+0000","last_became_peered":"2023-07-04T18:51:22.321426+0000","last_unstale":"2023-07-04T18:51:22.325268+0000","last_undegraded":"2023-07-04T18:51:22.325268+0000","last_fullsized":"2023-07-04T18:51:22.325268+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320229+0000","last_change":"2023-07-04T18:51:22.318535+0000","last_active":"2023-07-04T18:51:22.320229+0000","last_peered":"2023-07-04T18:51:22.320229+0000","last_clean":"2023-07-04T18:51:22.320229+0000","last_became_active":"2023-07-04T18:51:22.318160+0000","last_became_peered":"2023-07-04T18:51:22.318160+0000","last_unstale":"2023-07-04T18:51:22.320229+0000","last_undegraded":"2023-07-04T18:51:22.320229+0000","last_fullsized":"2023-07-04T18:51:22.320229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:44:26.146578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324837+0000","last_change":"2023-07-04T18:51:22.319854+0000","last_active":"2023-07-04T18:51:22.324837+0000","last_peered":"2023-07-04T18:51:22.324837+0000","last_clean":"2023-07-04T18:51:22.324837+0000","last_became_active":"2023-07-04T18:51:22.319581+0000","last_became_peered":"2023-07-04T18:51:22.319581+0000","last_unstale":"2023-07-04T18:51:22.324837+0000","last_undegraded":"2023-07-04T18:51:22.324837+0000","last_fullsized":"2023-07-04T18:51:22.324837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304943+0000","last_change":"2023-07-04T18:51:22.322222+0000","last_active":"2023-07-04T18:51:23.304943+0000","last_peered":"2023-07-04T18:51:23.304943+0000","last_clean":"2023-07-04T18:51:23.304943+0000","last_became_active":"2023-07-04T18:51:22.321944+0000","last_became_peered":"2023-07-04T18:51:22.321944+0000","last_unstale":"2023-07-04T18:51:23.304943+0000","last_undegraded":"2023-07-04T18:51:23.304943+0000","last_fullsized":"2023-07-04T18:51:23.304943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:53.251606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.322567+0000","last_change":"2023-07-04T18:51:22.318773+0000","last_active":"2023-07-04T18:51:22.322567+0000","last_peered":"2023-07-04T18:51:22.322567+0000","last_clean":"2023-07-04T18:51:22.322567+0000","last_became_active":"2023-07-04T18:51:22.318495+0000","last_became_peered":"2023-07-04T18:51:22.318495+0000","last_unstale":"2023-07-04T18:51:22.322567+0000","last_undegraded":"2023-07-04T18:51:22.322567+0000","last_fullsized":"2023-07-04T18:51:22.322567+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:19:18.779329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315977+0000","last_change":"2023-07-04T18:51:22.313313+0000","last_active":"2023-07-04T18:51:22.315977+0000","last_peered":"2023-07-04T18:51:22.315977+0000","last_clean":"2023-07-04T18:51:22.315977+0000","last_became_active":"2023-07-04T18:51:22.313049+0000","last_became_peered":"2023-07-04T18:51:22.313049+0000","last_unstale":"2023-07-04T18:51:22.315977+0000","last_undegraded":"2023-07-04T18:51:22.315977+0000","last_fullsized":"2023-07-04T18:51:22.315977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:31:55.772208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.325873+0000","last_change":"2023-07-04T18:51:22.322812+0000","last_active":"2023-07-04T18:51:22.325873+0000","last_peered":"2023-07-04T18:51:22.325873+0000","last_clean":"2023-07-04T18:51:22.325873+0000","last_became_active":"2023-07-04T18:51:22.318629+0000","last_became_peered":"2023-07-04T18:51:22.318629+0000","last_unstale":"2023-07-04T18:51:22.325873+0000","last_undegraded":"2023-07-04T18:51:22.325873+0000","last_fullsized":"2023-07-04T18:51:22.325873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450996+0000","last_change":"2023-07-04T18:51:22.323740+0000","last_active":"2023-07-04T18:51:23.450996+0000","last_peered":"2023-07-04T18:51:23.450996+0000","last_clean":"2023-07-04T18:51:23.450996+0000","last_became_active":"2023-07-04T18:51:22.323321+0000","last_became_peered":"2023-07-04T18:51:22.323321+0000","last_unstale":"2023-07-04T18:51:23.450996+0000","last_undegraded":"2023-07-04T18:51:23.450996+0000","last_fullsized":"2023-07-04T18:51:23.450996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:33:48.187398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304015+0000","last_change":"2023-07-04T18:51:22.318780+0000","last_active":"2023-07-04T18:51:23.304015+0000","last_peered":"2023-07-04T18:51:23.304015+0000","last_clean":"2023-07-04T18:51:23.304015+0000","last_became_active":"2023-07-04T18:51:22.318460+0000","last_became_peered":"2023-07-04T18:51:22.318460+0000","last_unstale":"2023-07-04T18:51:23.304015+0000","last_undegraded":"2023-07-04T18:51:23.304015+0000","last_fullsized":"2023-07-04T18:51:23.304015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320211+0000","last_change":"2023-07-04T18:51:22.318170+0000","last_active":"2023-07-04T18:51:22.320211+0000","last_peered":"2023-07-04T18:51:22.320211+0000","last_clean":"2023-07-04T18:51:22.320211+0000","last_became_active":"2023-07-04T18:51:22.317879+0000","last_became_peered":"2023-07-04T18:51:22.317879+0000","last_unstale":"2023-07-04T18:51:22.320211+0000","last_undegraded":"2023-07-04T18:51:22.320211+0000","last_fullsized":"2023-07-04T18:51:22.320211+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:07:21.587682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304493+0000","last_change":"2023-07-04T18:51:22.318559+0000","last_active":"2023-07-04T18:51:23.304493+0000","last_peered":"2023-07-04T18:51:23.304493+0000","last_clean":"2023-07-04T18:51:23.304493+0000","last_became_active":"2023-07-04T18:51:22.317739+0000","last_became_peered":"2023-07-04T18:51:22.317739+0000","last_unstale":"2023-07-04T18:51:23.304493+0000","last_undegraded":"2023-07-04T18:51:23.304493+0000","last_fullsized":"2023-07-04T18:51:23.304493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315900+0000","last_change":"2023-07-04T18:51:22.310959+0000","last_active":"2023-07-04T18:51:22.315900+0000","last_peered":"2023-07-04T18:51:22.315900+0000","last_clean":"2023-07-04T18:51:22.315900+0000","last_became_active":"2023-07-04T18:51:22.310668+0000","last_became_peered":"2023-07-04T18:51:22.310668+0000","last_unstale":"2023-07-04T18:51:22.315900+0000","last_undegraded":"2023-07-04T18:51:22.315900+0000","last_fullsized":"2023-07-04T18:51:22.315900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:44:26.146578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304398+0000","last_change":"2023-07-04T18:51:22.318672+0000","last_active":"2023-07-04T18:51:23.304398+0000","last_peered":"2023-07-04T18:51:23.304398+0000","last_clean":"2023-07-04T18:51:23.304398+0000","last_became_active":"2023-07-04T18:51:22.317856+0000","last_became_peered":"2023-07-04T18:51:22.317856+0000","last_unstale":"2023-07-04T18:51:23.304398+0000","last_undegraded":"2023-07-04T18:51:23.304398+0000","last_fullsized":"2023-07-04T18:51:23.304398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451144+0000","last_change":"2023-07-04T18:51:22.321788+0000","last_active":"2023-07-04T18:51:23.451144+0000","last_peered":"2023-07-04T18:51:23.451144+0000","last_clean":"2023-07-04T18:51:23.451144+0000","last_became_active":"2023-07-04T18:51:22.321394+0000","last_became_peered":"2023-07-04T18:51:22.321394+0000","last_unstale":"2023-07-04T18:51:23.451144+0000","last_undegraded":"2023-07-04T18:51:23.451144+0000","last_fullsized":"2023-07-04T18:51:23.451144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:00:25.437637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.323983+0000","last_change":"2023-07-04T18:51:22.321742+0000","last_active":"2023-07-04T18:51:22.323983+0000","last_peered":"2023-07-04T18:51:22.323983+0000","last_clean":"2023-07-04T18:51:22.323983+0000","last_became_active":"2023-07-04T18:51:22.321526+0000","last_became_peered":"2023-07-04T18:51:22.321526+0000","last_unstale":"2023-07-04T18:51:22.323983+0000","last_undegraded":"2023-07-04T18:51:22.323983+0000","last_fullsized":"2023-07-04T18:51:22.323983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:34:07.018645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303613+0000","last_change":"2023-07-04T18:51:22.321095+0000","last_active":"2023-07-04T18:51:23.303613+0000","last_peered":"2023-07-04T18:51:23.303613+0000","last_clean":"2023-07-04T18:51:23.303613+0000","last_became_active":"2023-07-04T18:51:22.320848+0000","last_became_peered":"2023-07-04T18:51:22.320848+0000","last_unstale":"2023-07-04T18:51:23.303613+0000","last_undegraded":"2023-07-04T18:51:23.303613+0000","last_fullsized":"2023-07-04T18:51:23.303613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450972+0000","last_change":"2023-07-04T18:51:22.322035+0000","last_active":"2023-07-04T18:51:23.450972+0000","last_peered":"2023-07-04T18:51:23.450972+0000","last_clean":"2023-07-04T18:51:23.450972+0000","last_became_active":"2023-07-04T18:51:22.321725+0000","last_became_peered":"2023-07-04T18:51:22.321725+0000","last_unstale":"2023-07-04T18:51:23.450972+0000","last_undegraded":"2023-07-04T18:51:23.450972+0000","last_fullsized":"2023-07-04T18:51:23.450972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417751+0000","last_change":"2023-07-04T18:51:22.311616+0000","last_active":"2023-07-04T18:51:23.417751+0000","last_peered":"2023-07-04T18:51:23.417751+0000","last_clean":"2023-07-04T18:51:23.417751+0000","last_became_active":"2023-07-04T18:51:22.311344+0000","last_became_peered":"2023-07-04T18:51:22.311344+0000","last_unstale":"2023-07-04T18:51:23.417751+0000","last_undegraded":"2023-07-04T18:51:23.417751+0000","last_fullsized":"2023-07-04T18:51:23.417751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417648+0000","last_change":"2023-07-04T18:51:22.311562+0000","last_active":"2023-07-04T18:51:23.417648+0000","last_peered":"2023-07-04T18:51:23.417648+0000","last_clean":"2023-07-04T18:51:23.417648+0000","last_became_active":"2023-07-04T18:51:22.311234+0000","last_became_peered":"2023-07-04T18:51:22.311234+0000","last_unstale":"2023-07-04T18:51:23.417648+0000","last_undegraded":"2023-07-04T18:51:23.417648+0000","last_fullsized":"2023-07-04T18:51:23.417648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.327028+0000","last_change":"2023-07-04T18:51:22.325120+0000","last_active":"2023-07-04T18:51:22.327028+0000","last_peered":"2023-07-04T18:51:22.327028+0000","last_clean":"2023-07-04T18:51:22.327028+0000","last_became_active":"2023-07-04T18:51:22.324834+0000","last_became_peered":"2023-07-04T18:51:22.324834+0000","last_unstale":"2023-07-04T18:51:22.327028+0000","last_undegraded":"2023-07-04T18:51:22.327028+0000","last_fullsized":"2023-07-04T18:51:22.327028+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304437+0000","last_change":"2023-07-04T18:51:21.460823+0000","last_active":"2023-07-04T18:51:23.304437+0000","last_peered":"2023-07-04T18:51:23.304437+0000","last_clean":"2023-07-04T18:51:23.304437+0000","last_became_active":"2023-07-04T18:51:21.460492+0000","last_became_peered":"2023-07-04T18:51:21.460492+0000","last_unstale":"2023-07-04T18:51:23.304437+0000","last_undegraded":"2023-07-04T18:51:23.304437+0000","last_fullsized":"2023-07-04T18:51:23.304437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303755+0000","last_change":"2023-07-04T18:51:22.313093+0000","last_active":"2023-07-04T18:51:23.303755+0000","last_peered":"2023-07-04T18:51:23.303755+0000","last_clean":"2023-07-04T18:51:23.303755+0000","last_became_active":"2023-07-04T18:51:22.312832+0000","last_became_peered":"2023-07-04T18:51:22.312832+0000","last_unstale":"2023-07-04T18:51:23.303755+0000","last_undegraded":"2023-07-04T18:51:23.303755+0000","last_fullsized":"2023-07-04T18:51:23.303755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417409+0000","last_change":"2023-07-04T18:51:21.471442+0000","last_active":"2023-07-04T18:51:23.417409+0000","last_peered":"2023-07-04T18:51:23.417409+0000","last_clean":"2023-07-04T18:51:23.417409+0000","last_became_active":"2023-07-04T18:51:21.471130+0000","last_became_peered":"2023-07-04T18:51:21.471130+0000","last_unstale":"2023-07-04T18:51:23.417409+0000","last_undegraded":"2023-07-04T18:51:23.417409+0000","last_fullsized":"2023-07-04T18:51:23.417409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:38.465858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324053+0000","last_change":"2023-07-04T18:51:22.322483+0000","last_active":"2023-07-04T18:51:22.324053+0000","last_peered":"2023-07-04T18:51:22.324053+0000","last_clean":"2023-07-04T18:51:22.324053+0000","last_became_active":"2023-07-04T18:51:22.322262+0000","last_became_peered":"2023-07-04T18:51:22.322262+0000","last_unstale":"2023-07-04T18:51:22.324053+0000","last_undegraded":"2023-07-04T18:51:22.324053+0000","last_fullsized":"2023-07-04T18:51:22.324053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:28:16.422678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303402+0000","last_change":"2023-07-04T18:51:21.470525+0000","last_active":"2023-07-04T18:51:23.303402+0000","last_peered":"2023-07-04T18:51:23.303402+0000","last_clean":"2023-07-04T18:51:23.303402+0000","last_became_active":"2023-07-04T18:51:21.470230+0000","last_became_peered":"2023-07-04T18:51:21.470230+0000","last_unstale":"2023-07-04T18:51:23.303402+0000","last_undegraded":"2023-07-04T18:51:23.303402+0000","last_fullsized":"2023-07-04T18:51:23.303402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303729+0000","last_change":"2023-07-04T18:51:22.323982+0000","last_active":"2023-07-04T18:51:23.303729+0000","last_peered":"2023-07-04T18:51:23.303729+0000","last_clean":"2023-07-04T18:51:23.303729+0000","last_became_active":"2023-07-04T18:51:22.323650+0000","last_became_peered":"2023-07-04T18:51:22.323650+0000","last_unstale":"2023-07-04T18:51:23.303729+0000","last_undegraded":"2023-07-04T18:51:23.303729+0000","last_fullsized":"2023-07-04T18:51:23.303729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309774+0000","last_change":"2023-07-04T18:51:21.464672+0000","last_active":"2023-07-04T18:51:22.309774+0000","last_peered":"2023-07-04T18:51:22.309774+0000","last_clean":"2023-07-04T18:51:22.309774+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:22.309774+0000","last_undegraded":"2023-07-04T18:51:22.309774+0000","last_fullsized":"2023-07-04T18:51:22.309774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:06.862561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303369+0000","last_change":"2023-07-04T18:51:22.311834+0000","last_active":"2023-07-04T18:51:23.303369+0000","last_peered":"2023-07-04T18:51:23.303369+0000","last_clean":"2023-07-04T18:51:23.303369+0000","last_became_active":"2023-07-04T18:51:22.311536+0000","last_became_peered":"2023-07-04T18:51:22.311536+0000","last_unstale":"2023-07-04T18:51:23.303369+0000","last_undegraded":"2023-07-04T18:51:23.303369+0000","last_fullsized":"2023-07-04T18:51:23.303369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.319384+0000","last_change":"2023-07-04T18:51:21.464692+0000","last_active":"2023-07-04T18:51:22.319384+0000","last_peered":"2023-07-04T18:51:22.319384+0000","last_clean":"2023-07-04T18:51:22.319384+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:22.319384+0000","last_undegraded":"2023-07-04T18:51:22.319384+0000","last_fullsized":"2023-07-04T18:51:22.319384+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303533+0000","last_change":"2023-07-04T18:51:22.320250+0000","last_active":"2023-07-04T18:51:23.303533+0000","last_peered":"2023-07-04T18:51:23.303533+0000","last_clean":"2023-07-04T18:51:23.303533+0000","last_became_active":"2023-07-04T18:51:22.319730+0000","last_became_peered":"2023-07-04T18:51:22.319730+0000","last_unstale":"2023-07-04T18:51:23.303533+0000","last_undegraded":"2023-07-04T18:51:23.303533+0000","last_fullsized":"2023-07-04T18:51:23.303533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451104+0000","last_change":"2023-07-04T18:51:21.471723+0000","last_active":"2023-07-04T18:51:23.451104+0000","last_peered":"2023-07-04T18:51:23.451104+0000","last_clean":"2023-07-04T18:51:23.451104+0000","last_became_active":"2023-07-04T18:51:21.471500+0000","last_became_peered":"2023-07-04T18:51:21.471500+0000","last_unstale":"2023-07-04T18:51:23.451104+0000","last_undegraded":"2023-07-04T18:51:23.451104+0000","last_fullsized":"2023-07-04T18:51:23.451104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:14:51.732204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417449+0000","last_change":"2023-07-04T18:51:22.311576+0000","last_active":"2023-07-04T18:51:23.417449+0000","last_peered":"2023-07-04T18:51:23.417449+0000","last_clean":"2023-07-04T18:51:23.417449+0000","last_became_active":"2023-07-04T18:51:22.311212+0000","last_became_peered":"2023-07-04T18:51:22.311212+0000","last_unstale":"2023-07-04T18:51:23.417449+0000","last_undegraded":"2023-07-04T18:51:23.417449+0000","last_fullsized":"2023-07-04T18:51:23.417449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304622+0000","last_change":"2023-07-04T18:51:21.464597+0000","last_active":"2023-07-04T18:51:23.304622+0000","last_peered":"2023-07-04T18:51:23.304622+0000","last_clean":"2023-07-04T18:51:23.304622+0000","last_became_active":"2023-07-04T18:51:21.464263+0000","last_became_peered":"2023-07-04T18:51:21.464263+0000","last_unstale":"2023-07-04T18:51:23.304622+0000","last_undegraded":"2023-07-04T18:51:23.304622+0000","last_fullsized":"2023-07-04T18:51:23.304622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304535+0000","last_change":"2023-07-04T18:51:22.318474+0000","last_active":"2023-07-04T18:51:23.304535+0000","last_peered":"2023-07-04T18:51:23.304535+0000","last_clean":"2023-07-04T18:51:23.304535+0000","last_became_active":"2023-07-04T18:51:22.317476+0000","last_became_peered":"2023-07-04T18:51:22.317476+0000","last_unstale":"2023-07-04T18:51:23.304535+0000","last_undegraded":"2023-07-04T18:51:23.304535+0000","last_fullsized":"2023-07-04T18:51:23.304535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450969+0000","last_change":"2023-07-04T18:51:21.467028+0000","last_active":"2023-07-04T18:51:23.450969+0000","last_peered":"2023-07-04T18:51:23.450969+0000","last_clean":"2023-07-04T18:51:23.450969+0000","last_became_active":"2023-07-04T18:51:21.466697+0000","last_became_peered":"2023-07-04T18:51:21.466697+0000","last_unstale":"2023-07-04T18:51:23.450969+0000","last_undegraded":"2023-07-04T18:51:23.450969+0000","last_fullsized":"2023-07-04T18:51:23.450969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303917+0000","last_change":"2023-07-04T18:51:22.320412+0000","last_active":"2023-07-04T18:51:23.303917+0000","last_peered":"2023-07-04T18:51:23.303917+0000","last_clean":"2023-07-04T18:51:23.303917+0000","last_became_active":"2023-07-04T18:51:22.320022+0000","last_became_peered":"2023-07-04T18:51:22.320022+0000","last_unstale":"2023-07-04T18:51:23.303917+0000","last_undegraded":"2023-07-04T18:51:23.303917+0000","last_fullsized":"2023-07-04T18:51:23.303917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309984+0000","last_change":"2023-07-04T18:51:21.465044+0000","last_active":"2023-07-04T18:51:22.309984+0000","last_peered":"2023-07-04T18:51:22.309984+0000","last_clean":"2023-07-04T18:51:22.309984+0000","last_became_active":"2023-07-04T18:51:21.464683+0000","last_became_peered":"2023-07-04T18:51:21.464683+0000","last_unstale":"2023-07-04T18:51:22.309984+0000","last_undegraded":"2023-07-04T18:51:22.309984+0000","last_fullsized":"2023-07-04T18:51:22.309984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:57.698962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.326905+0000","last_change":"2023-07-04T18:51:22.324545+0000","last_active":"2023-07-04T18:51:22.326905+0000","last_peered":"2023-07-04T18:51:22.326905+0000","last_clean":"2023-07-04T18:51:22.326905+0000","last_became_active":"2023-07-04T18:51:22.324168+0000","last_became_peered":"2023-07-04T18:51:22.324168+0000","last_unstale":"2023-07-04T18:51:22.326905+0000","last_undegraded":"2023-07-04T18:51:22.326905+0000","last_fullsized":"2023-07-04T18:51:22.326905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:20.924771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450810+0000","last_change":"2023-07-04T18:51:21.468878+0000","last_active":"2023-07-04T18:51:23.450810+0000","last_peered":"2023-07-04T18:51:23.450810+0000","last_clean":"2023-07-04T18:51:23.450810+0000","last_became_active":"2023-07-04T18:51:21.468229+0000","last_became_peered":"2023-07-04T18:51:21.468229+0000","last_unstale":"2023-07-04T18:51:23.450810+0000","last_undegraded":"2023-07-04T18:51:23.450810+0000","last_fullsized":"2023-07-04T18:51:23.450810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.320226+0000","last_change":"2023-07-04T18:51:22.317975+0000","last_active":"2023-07-04T18:51:22.320226+0000","last_peered":"2023-07-04T18:51:22.320226+0000","last_clean":"2023-07-04T18:51:22.320226+0000","last_became_active":"2023-07-04T18:51:22.317731+0000","last_became_peered":"2023-07-04T18:51:22.317731+0000","last_unstale":"2023-07-04T18:51:22.320226+0000","last_undegraded":"2023-07-04T18:51:22.320226+0000","last_fullsized":"2023-07-04T18:51:22.320226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303059+0000","last_change":"2023-07-04T18:51:21.460954+0000","last_active":"2023-07-04T18:51:23.303059+0000","last_peered":"2023-07-04T18:51:23.303059+0000","last_clean":"2023-07-04T18:51:23.303059+0000","last_became_active":"2023-07-04T18:51:21.460663+0000","last_became_peered":"2023-07-04T18:51:21.460663+0000","last_unstale":"2023-07-04T18:51:23.303059+0000","last_undegraded":"2023-07-04T18:51:23.303059+0000","last_fullsized":"2023-07-04T18:51:23.303059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:08.443319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304053+0000","last_change":"2023-07-04T18:51:22.313798+0000","last_active":"2023-07-04T18:51:23.304053+0000","last_peered":"2023-07-04T18:51:23.304053+0000","last_clean":"2023-07-04T18:51:23.304053+0000","last_became_active":"2023-07-04T18:51:22.313526+0000","last_became_peered":"2023-07-04T18:51:22.313526+0000","last_unstale":"2023-07-04T18:51:23.304053+0000","last_undegraded":"2023-07-04T18:51:23.304053+0000","last_fullsized":"2023-07-04T18:51:23.304053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451341+0000","last_change":"2023-07-04T18:51:21.468130+0000","last_active":"2023-07-04T18:51:23.451341+0000","last_peered":"2023-07-04T18:51:23.451341+0000","last_clean":"2023-07-04T18:51:23.451341+0000","last_became_active":"2023-07-04T18:51:21.467832+0000","last_became_peered":"2023-07-04T18:51:21.467832+0000","last_unstale":"2023-07-04T18:51:23.451341+0000","last_undegraded":"2023-07-04T18:51:23.451341+0000","last_fullsized":"2023-07-04T18:51:23.451341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:02:58.666739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.327043+0000","last_change":"2023-07-04T18:51:22.325399+0000","last_active":"2023-07-04T18:51:22.327043+0000","last_peered":"2023-07-04T18:51:22.327043+0000","last_clean":"2023-07-04T18:51:22.327043+0000","last_became_active":"2023-07-04T18:51:22.324669+0000","last_became_peered":"2023-07-04T18:51:22.324669+0000","last_unstale":"2023-07-04T18:51:22.327043+0000","last_undegraded":"2023-07-04T18:51:22.327043+0000","last_fullsized":"2023-07-04T18:51:22.327043+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:18.636942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451202+0000","last_change":"2023-07-04T18:51:21.466954+0000","last_active":"2023-07-04T18:51:23.451202+0000","last_peered":"2023-07-04T18:51:23.451202+0000","last_clean":"2023-07-04T18:51:23.451202+0000","last_became_active":"2023-07-04T18:51:21.466625+0000","last_became_peered":"2023-07-04T18:51:21.466625+0000","last_unstale":"2023-07-04T18:51:23.451202+0000","last_undegraded":"2023-07-04T18:51:23.451202+0000","last_fullsized":"2023-07-04T18:51:23.451202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:17:35.188150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304676+0000","last_change":"2023-07-04T18:51:22.321314+0000","last_active":"2023-07-04T18:51:23.304676+0000","last_peered":"2023-07-04T18:51:23.304676+0000","last_clean":"2023-07-04T18:51:23.304676+0000","last_became_active":"2023-07-04T18:51:22.321008+0000","last_became_peered":"2023-07-04T18:51:22.321008+0000","last_unstale":"2023-07-04T18:51:23.304676+0000","last_undegraded":"2023-07-04T18:51:23.304676+0000","last_fullsized":"2023-07-04T18:51:23.304676+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451276+0000","last_change":"2023-07-04T18:51:21.468101+0000","last_active":"2023-07-04T18:51:23.451276+0000","last_peered":"2023-07-04T18:51:23.451276+0000","last_clean":"2023-07-04T18:51:23.451276+0000","last_became_active":"2023-07-04T18:51:21.467813+0000","last_became_peered":"2023-07-04T18:51:21.467813+0000","last_unstale":"2023-07-04T18:51:23.451276+0000","last_undegraded":"2023-07-04T18:51:23.451276+0000","last_fullsized":"2023-07-04T18:51:23.451276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:02:10.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303594+0000","last_change":"2023-07-04T18:51:22.324752+0000","last_active":"2023-07-04T18:51:23.303594+0000","last_peered":"2023-07-04T18:51:23.303594+0000","last_clean":"2023-07-04T18:51:23.303594+0000","last_became_active":"2023-07-04T18:51:22.324420+0000","last_became_peered":"2023-07-04T18:51:22.324420+0000","last_unstale":"2023-07-04T18:51:23.303594+0000","last_undegraded":"2023-07-04T18:51:23.303594+0000","last_fullsized":"2023-07-04T18:51:23.303594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:06.453079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316725+0000","last_change":"2023-07-04T18:51:21.471602+0000","last_active":"2023-07-04T18:51:22.316725+0000","last_peered":"2023-07-04T18:51:22.316725+0000","last_clean":"2023-07-04T18:51:22.316725+0000","last_became_active":"2023-07-04T18:51:21.471319+0000","last_became_peered":"2023-07-04T18:51:21.471319+0000","last_unstale":"2023-07-04T18:51:22.316725+0000","last_undegraded":"2023-07-04T18:51:22.316725+0000","last_fullsized":"2023-07-04T18:51:22.316725+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:29.652241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.323916+0000","last_change":"2023-07-04T18:51:22.321087+0000","last_active":"2023-07-04T18:51:22.323916+0000","last_peered":"2023-07-04T18:51:22.323916+0000","last_clean":"2023-07-04T18:51:22.323916+0000","last_became_active":"2023-07-04T18:51:22.320871+0000","last_became_peered":"2023-07-04T18:51:22.320871+0000","last_unstale":"2023-07-04T18:51:22.323916+0000","last_undegraded":"2023-07-04T18:51:22.323916+0000","last_fullsized":"2023-07-04T18:51:22.323916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:27.753990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.473477+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.473173+0000","last_became_peered":"2023-07-04T18:51:21.473173+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304599+0000","last_change":"2023-07-04T18:51:22.317634+0000","last_active":"2023-07-04T18:51:23.304599+0000","last_peered":"2023-07-04T18:51:23.304599+0000","last_clean":"2023-07-04T18:51:23.304599+0000","last_became_active":"2023-07-04T18:51:22.317260+0000","last_became_peered":"2023-07-04T18:51:22.317260+0000","last_unstale":"2023-07-04T18:51:23.304599+0000","last_undegraded":"2023-07-04T18:51:23.304599+0000","last_fullsized":"2023-07-04T18:51:23.304599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303472+0000","last_change":"2023-07-04T18:51:21.463128+0000","last_active":"2023-07-04T18:51:23.303472+0000","last_peered":"2023-07-04T18:51:23.303472+0000","last_clean":"2023-07-04T18:51:23.303472+0000","last_became_active":"2023-07-04T18:51:21.462860+0000","last_became_peered":"2023-07-04T18:51:21.462860+0000","last_unstale":"2023-07-04T18:51:23.303472+0000","last_undegraded":"2023-07-04T18:51:23.303472+0000","last_fullsized":"2023-07-04T18:51:23.303472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304928+0000","last_change":"2023-07-04T18:51:22.318985+0000","last_active":"2023-07-04T18:51:23.304928+0000","last_peered":"2023-07-04T18:51:23.304928+0000","last_clean":"2023-07-04T18:51:23.304928+0000","last_became_active":"2023-07-04T18:51:22.318393+0000","last_became_peered":"2023-07-04T18:51:22.318393+0000","last_unstale":"2023-07-04T18:51:23.304928+0000","last_undegraded":"2023-07-04T18:51:23.304928+0000","last_fullsized":"2023-07-04T18:51:23.304928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304517+0000","last_change":"2023-07-04T18:51:21.466769+0000","last_active":"2023-07-04T18:51:23.304517+0000","last_peered":"2023-07-04T18:51:23.304517+0000","last_clean":"2023-07-04T18:51:23.304517+0000","last_became_active":"2023-07-04T18:51:21.466504+0000","last_became_peered":"2023-07-04T18:51:21.466504+0000","last_unstale":"2023-07-04T18:51:23.304517+0000","last_undegraded":"2023-07-04T18:51:23.304517+0000","last_fullsized":"2023-07-04T18:51:23.304517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:03.327143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304757+0000","last_change":"2023-07-04T18:51:22.317855+0000","last_active":"2023-07-04T18:51:23.304757+0000","last_peered":"2023-07-04T18:51:23.304757+0000","last_clean":"2023-07-04T18:51:23.304757+0000","last_became_active":"2023-07-04T18:51:22.317426+0000","last_became_peered":"2023-07-04T18:51:22.317426+0000","last_unstale":"2023-07-04T18:51:23.304757+0000","last_undegraded":"2023-07-04T18:51:23.304757+0000","last_fullsized":"2023-07-04T18:51:23.304757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303314+0000","last_change":"2023-07-04T18:51:21.471306+0000","last_active":"2023-07-04T18:51:23.303314+0000","last_peered":"2023-07-04T18:51:23.303314+0000","last_clean":"2023-07-04T18:51:23.303314+0000","last_became_active":"2023-07-04T18:51:21.471034+0000","last_became_peered":"2023-07-04T18:51:21.471034+0000","last_unstale":"2023-07-04T18:51:23.303314+0000","last_undegraded":"2023-07-04T18:51:23.303314+0000","last_fullsized":"2023-07-04T18:51:23.303314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417826+0000","last_change":"2023-07-04T18:51:22.315022+0000","last_active":"2023-07-04T18:51:23.417826+0000","last_peered":"2023-07-04T18:51:23.417826+0000","last_clean":"2023-07-04T18:51:23.417826+0000","last_became_active":"2023-07-04T18:51:22.314767+0000","last_became_peered":"2023-07-04T18:51:22.314767+0000","last_unstale":"2023-07-04T18:51:23.417826+0000","last_undegraded":"2023-07-04T18:51:23.417826+0000","last_fullsized":"2023-07-04T18:51:23.417826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417511+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:51:23.417511+0000","last_peered":"2023-07-04T18:51:23.417511+0000","last_clean":"2023-07-04T18:51:23.417511+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:51:23.417511+0000","last_undegraded":"2023-07-04T18:51:23.417511+0000","last_fullsized":"2023-07-04T18:51:23.417511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.317007+0000","last_change":"2023-07-04T18:51:21.471879+0000","last_active":"2023-07-04T18:51:22.317007+0000","last_peered":"2023-07-04T18:51:22.317007+0000","last_clean":"2023-07-04T18:51:22.317007+0000","last_became_active":"2023-07-04T18:51:21.471549+0000","last_became_peered":"2023-07-04T18:51:21.471549+0000","last_unstale":"2023-07-04T18:51:22.317007+0000","last_undegraded":"2023-07-04T18:51:22.317007+0000","last_fullsized":"2023-07-04T18:51:22.317007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:50.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417554+0000","last_change":"2023-07-04T18:51:22.319180+0000","last_active":"2023-07-04T18:51:23.417554+0000","last_peered":"2023-07-04T18:51:23.417554+0000","last_clean":"2023-07-04T18:51:23.417554+0000","last_became_active":"2023-07-04T18:51:22.318906+0000","last_became_peered":"2023-07-04T18:51:22.318906+0000","last_unstale":"2023-07-04T18:51:23.417554+0000","last_undegraded":"2023-07-04T18:51:23.417554+0000","last_fullsized":"2023-07-04T18:51:23.417554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316574+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:51:22.316574+0000","last_peered":"2023-07-04T18:51:22.316574+0000","last_clean":"2023-07-04T18:51:22.316574+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:51:22.316574+0000","last_undegraded":"2023-07-04T18:51:22.316574+0000","last_fullsized":"2023-07-04T18:51:22.316574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:01.315083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304959+0000","last_change":"2023-07-04T18:51:21.461532+0000","last_active":"2023-07-04T18:51:23.304959+0000","last_peered":"2023-07-04T18:51:23.304959+0000","last_clean":"2023-07-04T18:51:23.304959+0000","last_became_active":"2023-07-04T18:51:21.460816+0000","last_became_peered":"2023-07-04T18:51:21.460816+0000","last_unstale":"2023-07-04T18:51:23.304959+0000","last_undegraded":"2023-07-04T18:51:23.304959+0000","last_fullsized":"2023-07-04T18:51:23.304959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417715+0000","last_change":"2023-07-04T18:51:22.312480+0000","last_active":"2023-07-04T18:51:23.417715+0000","last_peered":"2023-07-04T18:51:23.417715+0000","last_clean":"2023-07-04T18:51:23.417715+0000","last_became_active":"2023-07-04T18:51:22.311193+0000","last_became_peered":"2023-07-04T18:51:22.311193+0000","last_unstale":"2023-07-04T18:51:23.417715+0000","last_undegraded":"2023-07-04T18:51:23.417715+0000","last_fullsized":"2023-07-04T18:51:23.417715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.301750+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:51:22.301750+0000","last_peered":"2023-07-04T18:51:22.301750+0000","last_clean":"2023-07-04T18:51:22.301750+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:51:22.301750+0000","last_undegraded":"2023-07-04T18:51:22.301750+0000","last_fullsized":"2023-07-04T18:51:22.301750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:36:47.185435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417530+0000","last_change":"2023-07-04T18:51:21.467449+0000","last_active":"2023-07-04T18:51:23.417530+0000","last_peered":"2023-07-04T18:51:23.417530+0000","last_clean":"2023-07-04T18:51:23.417530+0000","last_became_active":"2023-07-04T18:51:21.466522+0000","last_became_peered":"2023-07-04T18:51:21.466522+0000","last_unstale":"2023-07-04T18:51:23.417530+0000","last_undegraded":"2023-07-04T18:51:23.417530+0000","last_fullsized":"2023-07-04T18:51:23.417530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417692+0000","last_change":"2023-07-04T18:51:22.324258+0000","last_active":"2023-07-04T18:51:23.417692+0000","last_peered":"2023-07-04T18:51:23.417692+0000","last_clean":"2023-07-04T18:51:23.417692+0000","last_became_active":"2023-07-04T18:51:22.323966+0000","last_became_peered":"2023-07-04T18:51:22.323966+0000","last_unstale":"2023-07-04T18:51:23.417692+0000","last_undegraded":"2023-07-04T18:51:23.417692+0000","last_fullsized":"2023-07-04T18:51:23.417692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.316509+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:51:22.316509+0000","last_peered":"2023-07-04T18:51:22.316509+0000","last_clean":"2023-07-04T18:51:22.316509+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:51:22.316509+0000","last_undegraded":"2023-07-04T18:51:22.316509+0000","last_fullsized":"2023-07-04T18:51:22.316509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450698+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:51:23.450698+0000","last_peered":"2023-07-04T18:51:23.450698+0000","last_clean":"2023-07-04T18:51:23.450698+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:51:23.450698+0000","last_undegraded":"2023-07-04T18:51:23.450698+0000","last_fullsized":"2023-07-04T18:51:23.450698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:37.197069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303756+0000","last_change":"2023-07-04T18:51:21.470626+0000","last_active":"2023-07-04T18:51:23.303756+0000","last_peered":"2023-07-04T18:51:23.303756+0000","last_clean":"2023-07-04T18:51:23.303756+0000","last_became_active":"2023-07-04T18:51:21.470351+0000","last_became_peered":"2023-07-04T18:51:21.470351+0000","last_unstale":"2023-07-04T18:51:23.303756+0000","last_undegraded":"2023-07-04T18:51:23.303756+0000","last_fullsized":"2023-07-04T18:51:23.303756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303308+0000","last_change":"2023-07-04T18:51:22.321449+0000","last_active":"2023-07-04T18:51:23.303308+0000","last_peered":"2023-07-04T18:51:23.303308+0000","last_clean":"2023-07-04T18:51:23.303308+0000","last_became_active":"2023-07-04T18:51:22.321158+0000","last_became_peered":"2023-07-04T18:51:22.321158+0000","last_unstale":"2023-07-04T18:51:23.303308+0000","last_undegraded":"2023-07-04T18:51:23.303308+0000","last_fullsized":"2023-07-04T18:51:23.303308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450897+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:51:23.450897+0000","last_peered":"2023-07-04T18:51:23.450897+0000","last_clean":"2023-07-04T18:51:23.450897+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:51:23.450897+0000","last_undegraded":"2023-07-04T18:51:23.450897+0000","last_fullsized":"2023-07-04T18:51:23.450897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:48.113610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309128+0000","last_change":"2023-07-04T18:51:21.460314+0000","last_active":"2023-07-04T18:51:22.309128+0000","last_peered":"2023-07-04T18:51:22.309128+0000","last_clean":"2023-07-04T18:51:22.309128+0000","last_became_active":"2023-07-04T18:51:21.460022+0000","last_became_peered":"2023-07-04T18:51:21.460022+0000","last_unstale":"2023-07-04T18:51:22.309128+0000","last_undegraded":"2023-07-04T18:51:22.309128+0000","last_fullsized":"2023-07-04T18:51:22.309128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:17.635315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.324037+0000","last_change":"2023-07-04T18:51:22.321571+0000","last_active":"2023-07-04T18:51:22.324037+0000","last_peered":"2023-07-04T18:51:22.324037+0000","last_clean":"2023-07-04T18:51:22.324037+0000","last_became_active":"2023-07-04T18:51:22.321326+0000","last_became_peered":"2023-07-04T18:51:22.321326+0000","last_unstale":"2023-07-04T18:51:22.324037+0000","last_undegraded":"2023-07-04T18:51:22.324037+0000","last_fullsized":"2023-07-04T18:51:22.324037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:20.924771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451151+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:51:23.451151+0000","last_peered":"2023-07-04T18:51:23.451151+0000","last_clean":"2023-07-04T18:51:23.451151+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:51:23.451151+0000","last_undegraded":"2023-07-04T18:51:23.451151+0000","last_fullsized":"2023-07-04T18:51:23.451151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:14:08.639700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417718+0000","last_change":"2023-07-04T18:51:21.466845+0000","last_active":"2023-07-04T18:51:23.417718+0000","last_peered":"2023-07-04T18:51:23.417718+0000","last_clean":"2023-07-04T18:51:23.417718+0000","last_became_active":"2023-07-04T18:51:21.466546+0000","last_became_peered":"2023-07-04T18:51:21.466546+0000","last_unstale":"2023-07-04T18:51:23.417718+0000","last_undegraded":"2023-07-04T18:51:23.417718+0000","last_fullsized":"2023-07-04T18:51:23.417718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.321659+0000","last_change":"2023-07-04T18:51:22.318511+0000","last_active":"2023-07-04T18:51:22.321659+0000","last_peered":"2023-07-04T18:51:22.321659+0000","last_clean":"2023-07-04T18:51:22.321659+0000","last_became_active":"2023-07-04T18:51:22.318227+0000","last_became_peered":"2023-07-04T18:51:22.318227+0000","last_unstale":"2023-07-04T18:51:22.321659+0000","last_undegraded":"2023-07-04T18:51:22.321659+0000","last_fullsized":"2023-07-04T18:51:22.321659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:31:55.772208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304031+0000","last_change":"2023-07-04T18:51:21.474661+0000","last_active":"2023-07-04T18:51:23.304031+0000","last_peered":"2023-07-04T18:51:23.304031+0000","last_clean":"2023-07-04T18:51:23.304031+0000","last_became_active":"2023-07-04T18:51:21.474400+0000","last_became_peered":"2023-07-04T18:51:21.474400+0000","last_unstale":"2023-07-04T18:51:23.304031+0000","last_undegraded":"2023-07-04T18:51:23.304031+0000","last_fullsized":"2023-07-04T18:51:23.304031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417697+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:51:23.417697+0000","last_peered":"2023-07-04T18:51:23.417697+0000","last_clean":"2023-07-04T18:51:23.417697+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:51:23.417697+0000","last_undegraded":"2023-07-04T18:51:23.417697+0000","last_fullsized":"2023-07-04T18:51:23.417697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:02.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303243+0000","last_change":"2023-07-04T18:51:22.313754+0000","last_active":"2023-07-04T18:51:23.303243+0000","last_peered":"2023-07-04T18:51:23.303243+0000","last_clean":"2023-07-04T18:51:23.303243+0000","last_became_active":"2023-07-04T18:51:22.313509+0000","last_became_peered":"2023-07-04T18:51:22.313509+0000","last_unstale":"2023-07-04T18:51:23.303243+0000","last_undegraded":"2023-07-04T18:51:23.303243+0000","last_fullsized":"2023-07-04T18:51:23.303243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303419+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:51:23.303419+0000","last_peered":"2023-07-04T18:51:23.303419+0000","last_clean":"2023-07-04T18:51:23.303419+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:51:23.303419+0000","last_undegraded":"2023-07-04T18:51:23.303419+0000","last_fullsized":"2023-07-04T18:51:23.303419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.309923+0000","last_change":"2023-07-04T18:51:21.464739+0000","last_active":"2023-07-04T18:51:22.309923+0000","last_peered":"2023-07-04T18:51:22.309923+0000","last_clean":"2023-07-04T18:51:22.309923+0000","last_became_active":"2023-07-04T18:51:21.464445+0000","last_became_peered":"2023-07-04T18:51:21.464445+0000","last_unstale":"2023-07-04T18:51:22.309923+0000","last_undegraded":"2023-07-04T18:51:22.309923+0000","last_fullsized":"2023-07-04T18:51:22.309923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:43.919530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304372+0000","last_change":"2023-07-04T18:51:22.326679+0000","last_active":"2023-07-04T18:51:23.304372+0000","last_peered":"2023-07-04T18:51:23.304372+0000","last_clean":"2023-07-04T18:51:23.304372+0000","last_became_active":"2023-07-04T18:51:22.326348+0000","last_became_peered":"2023-07-04T18:51:22.326348+0000","last_unstale":"2023-07-04T18:51:23.304372+0000","last_undegraded":"2023-07-04T18:51:23.304372+0000","last_fullsized":"2023-07-04T18:51:23.304372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.486397+0000","last_change":"2023-07-04T18:51:21.466007+0000","last_active":"2023-07-04T18:51:22.486397+0000","last_peered":"2023-07-04T18:51:22.486397+0000","last_clean":"2023-07-04T18:51:22.486397+0000","last_became_active":"2023-07-04T18:51:21.465754+0000","last_became_peered":"2023-07-04T18:51:21.465754+0000","last_unstale":"2023-07-04T18:51:22.486397+0000","last_undegraded":"2023-07-04T18:51:22.486397+0000","last_fullsized":"2023-07-04T18:51:22.486397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:41.484807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.315223+0000","last_change":"2023-07-04T18:51:22.309513+0000","last_active":"2023-07-04T18:51:22.315223+0000","last_peered":"2023-07-04T18:51:22.315223+0000","last_clean":"2023-07-04T18:51:22.315223+0000","last_became_active":"2023-07-04T18:51:22.309252+0000","last_became_peered":"2023-07-04T18:51:22.309252+0000","last_unstale":"2023-07-04T18:51:22.315223+0000","last_undegraded":"2023-07-04T18:51:22.315223+0000","last_fullsized":"2023-07-04T18:51:22.315223+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:34:07.018645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.486707+0000","last_change":"2023-07-04T18:51:21.463261+0000","last_active":"2023-07-04T18:51:22.486707+0000","last_peered":"2023-07-04T18:51:22.486707+0000","last_clean":"2023-07-04T18:51:22.486707+0000","last_became_active":"2023-07-04T18:51:21.462985+0000","last_became_peered":"2023-07-04T18:51:21.462985+0000","last_unstale":"2023-07-04T18:51:22.486707+0000","last_undegraded":"2023-07-04T18:51:22.486707+0000","last_fullsized":"2023-07-04T18:51:22.486707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304859+0000","last_change":"2023-07-04T18:51:22.318260+0000","last_active":"2023-07-04T18:51:23.304859+0000","last_peered":"2023-07-04T18:51:23.304859+0000","last_clean":"2023-07-04T18:51:23.304859+0000","last_became_active":"2023-07-04T18:51:22.317037+0000","last_became_peered":"2023-07-04T18:51:22.317037+0000","last_unstale":"2023-07-04T18:51:23.304859+0000","last_undegraded":"2023-07-04T18:51:23.304859+0000","last_fullsized":"2023-07-04T18:51:23.304859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.487409+0000","last_change":"2023-07-04T18:51:21.463234+0000","last_active":"2023-07-04T18:51:22.487409+0000","last_peered":"2023-07-04T18:51:22.487409+0000","last_clean":"2023-07-04T18:51:22.487409+0000","last_became_active":"2023-07-04T18:51:21.462954+0000","last_became_peered":"2023-07-04T18:51:21.462954+0000","last_unstale":"2023-07-04T18:51:22.487409+0000","last_undegraded":"2023-07-04T18:51:22.487409+0000","last_fullsized":"2023-07-04T18:51:22.487409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:09.767759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304612+0000","last_change":"2023-07-04T18:51:22.317628+0000","last_active":"2023-07-04T18:51:23.304612+0000","last_peered":"2023-07-04T18:51:23.304612+0000","last_clean":"2023-07-04T18:51:23.304612+0000","last_became_active":"2023-07-04T18:51:22.317043+0000","last_became_peered":"2023-07-04T18:51:22.317043+0000","last_unstale":"2023-07-04T18:51:23.304612+0000","last_undegraded":"2023-07-04T18:51:23.304612+0000","last_fullsized":"2023-07-04T18:51:23.304612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.470557+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.470247+0000","last_became_peered":"2023-07-04T18:51:21.470247+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:13.171949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303469+0000","last_change":"2023-07-04T18:51:22.313146+0000","last_active":"2023-07-04T18:51:23.303469+0000","last_peered":"2023-07-04T18:51:23.303469+0000","last_clean":"2023-07-04T18:51:23.303469+0000","last_became_active":"2023-07-04T18:51:22.312877+0000","last_became_peered":"2023-07-04T18:51:22.312877+0000","last_unstale":"2023-07-04T18:51:23.303469+0000","last_undegraded":"2023-07-04T18:51:23.303469+0000","last_fullsized":"2023-07-04T18:51:23.303469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-04T18:51:22.489888+0000","last_change":"2023-07-04T18:51:21.459772+0000","last_active":"2023-07-04T18:51:22.489888+0000","last_peered":"2023-07-04T18:51:22.489888+0000","last_clean":"2023-07-04T18:51:22.489888+0000","last_became_active":"2023-07-04T18:51:21.459459+0000","last_became_peered":"2023-07-04T18:51:21.459459+0000","last_unstale":"2023-07-04T18:51:22.489888+0000","last_undegraded":"2023-07-04T18:51:22.489888+0000","last_fullsized":"2023-07-04T18:51:22.489888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303922+0000","last_change":"2023-07-04T18:51:22.318229+0000","last_active":"2023-07-04T18:51:23.303922+0000","last_peered":"2023-07-04T18:51:23.303922+0000","last_clean":"2023-07-04T18:51:23.303922+0000","last_became_active":"2023-07-04T18:51:22.317388+0000","last_became_peered":"2023-07-04T18:51:22.317388+0000","last_unstale":"2023-07-04T18:51:23.303922+0000","last_undegraded":"2023-07-04T18:51:23.303922+0000","last_fullsized":"2023-07-04T18:51:23.303922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:55.968814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303155+0000","last_change":"2023-07-04T18:51:22.314933+0000","last_active":"2023-07-04T18:51:23.303155+0000","last_peered":"2023-07-04T18:51:23.303155+0000","last_clean":"2023-07-04T18:51:23.303155+0000","last_became_active":"2023-07-04T18:51:22.314648+0000","last_became_peered":"2023-07-04T18:51:22.314648+0000","last_unstale":"2023-07-04T18:51:23.303155+0000","last_undegraded":"2023-07-04T18:51:23.303155+0000","last_fullsized":"2023-07-04T18:51:23.303155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303365+0000","last_change":"2023-07-04T18:51:22.318422+0000","last_active":"2023-07-04T18:51:23.303365+0000","last_peered":"2023-07-04T18:51:23.303365+0000","last_clean":"2023-07-04T18:51:23.303365+0000","last_became_active":"2023-07-04T18:51:22.318161+0000","last_became_peered":"2023-07-04T18:51:22.318161+0000","last_unstale":"2023-07-04T18:51:23.303365+0000","last_undegraded":"2023-07-04T18:51:23.303365+0000","last_fullsized":"2023-07-04T18:51:23.303365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304314+0000","last_change":"2023-07-04T18:51:22.320144+0000","last_active":"2023-07-04T18:51:23.304314+0000","last_peered":"2023-07-04T18:51:23.304314+0000","last_clean":"2023-07-04T18:51:23.304314+0000","last_became_active":"2023-07-04T18:51:22.317892+0000","last_became_peered":"2023-07-04T18:51:22.317892+0000","last_unstale":"2023-07-04T18:51:23.304314+0000","last_undegraded":"2023-07-04T18:51:23.304314+0000","last_fullsized":"2023-07-04T18:51:23.304314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303934+0000","last_change":"2023-07-04T18:51:22.317154+0000","last_active":"2023-07-04T18:51:23.303934+0000","last_peered":"2023-07-04T18:51:23.303934+0000","last_clean":"2023-07-04T18:51:23.303934+0000","last_became_active":"2023-07-04T18:51:22.316916+0000","last_became_peered":"2023-07-04T18:51:22.316916+0000","last_unstale":"2023-07-04T18:51:23.303934+0000","last_undegraded":"2023-07-04T18:51:23.303934+0000","last_fullsized":"2023-07-04T18:51:23.303934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417430+0000","last_change":"2023-07-04T18:51:22.312307+0000","last_active":"2023-07-04T18:51:23.417430+0000","last_peered":"2023-07-04T18:51:23.417430+0000","last_clean":"2023-07-04T18:51:23.417430+0000","last_became_active":"2023-07-04T18:51:22.310832+0000","last_became_peered":"2023-07-04T18:51:22.310832+0000","last_unstale":"2023-07-04T18:51:23.417430+0000","last_undegraded":"2023-07-04T18:51:23.417430+0000","last_fullsized":"2023-07-04T18:51:23.417430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":864256,"data_stored":681747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":827392,"data_stored":673896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":229376,"data_stored":82946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":262144,"data_stored":91219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":237568,"data_stored":83598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:51:26.954 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-04T18:51:26.954 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-04T18:51:26.954 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-07-04T18:51:26.955 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-07-04T18:51:26.955 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-07-04T18:51:27.107 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.106+0000 7f1de8730700 1 Processor -- start 2023-07-04T18:51:27.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.106+0000 7f1de8730700 1 -- start start 2023-07-04T18:51:27.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de8730700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f1de015c640 0x7f1de00cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de8730700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00717f0 con 0x7f1de0070f80 2023-07-04T18:51:27.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de8730700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de00cd290 con 0x7f1de015ca10 2023-07-04T18:51:27.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de8730700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00ccfb0 con 0x7f1de015c640 2023-07-04T18:51:27.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de6ccd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1de0070f80 0x7f1de00cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51078/0 (socket says 172.21.15.77:51078) 2023-07-04T18:51:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.107+0000 7f1de6ccd700 1 -- 172.21.15.77:0/3553297691 learned_addr learned my addr 172.21.15.77:0/3553297691 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de64cc700 1 --2- 172.21.15.77:0/3553297691 >> v2:172.21.15.82:3300/0 conn(0x7f1de015c640 0x7f1de00cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047856906 0 0) 0x7f1de00717f0 con 0x7f1de0070f80 2023-07-04T18:51:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc0035b0 con 0x7f1de0070f80 2023-07-04T18:51:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de64cc700 1 -- 172.21.15.77:0/3553297691 >> v1:172.21.15.82:6789/0 conn(0x7f1de015ca10 legacy=0x7f1de0070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634266499 0 0) 0x7f1dcc0035b0 con 0x7f1de0070f80 2023-07-04T18:51:27.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de64cc700 1 -- 172.21.15.77:0/3553297691 >> v1:172.21.15.77:6789/0 conn(0x7f1de0070f80 legacy=0x7f1de00cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de64cc700 1 -- 172.21.15.77:0/3553297691 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00ce640 con 0x7f1de015c640 2023-07-04T18:51:27.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de64cc700 1 --2- 172.21.15.77:0/3553297691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de015c640 0x7f1de00cf120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7709f95683b0f31 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80019f0 con 0x7f1de015c640 2023-07-04T18:51:27.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0004030 con 0x7f1de015c640 2023-07-04T18:51:27.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.108+0000 7f1de54ca700 1 -- 172.21.15.77:0/3553297691 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00043a0 con 0x7f1de015c640 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 -- 172.21.15.77:0/3553297691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de015c640 msgr2=0x7f1de00cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 --2- 172.21.15.77:0/3553297691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de015c640 0x7f1de00cf120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009840 tx=0x7f1dd0009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 -- 172.21.15.77:0/3553297691 shutdown_connections 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 --2- 172.21.15.77:0/3553297691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de015c640 0x7f1de00cf120 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 -- 172.21.15.77:0/3553297691 >> 172.21.15.77:0/3553297691 conn(0x7f1de006bba0 msgr2=0x7f1de00b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:27.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 -- 172.21.15.77:0/3553297691 shutdown_connections 2023-07-04T18:51:27.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.109+0000 7f1de8730700 1 -- 172.21.15.77:0/3553297691 wait complete. 2023-07-04T18:51:27.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.110+0000 7f1de8730700 1 Processor -- start 2023-07-04T18:51:27.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.110+0000 7f1de8730700 1 -- start start 2023-07-04T18:51:27.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de0070f80 0x7f1de0148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1de015c640 0x7f1de0148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00c8b30 con 0x7f1de015ca10 2023-07-04T18:51:27.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00c99c0 con 0x7f1de0070f80 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00c8db0 con 0x7f1de015c640 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43946/0 (socket says 172.21.15.77:43946) 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 -- 172.21.15.77:0/1453137977 learned_addr learned my addr 172.21.15.77:0/1453137977 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de64cc700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de0070f80 0x7f1de0148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de5ccb700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1de015c640 0x7f1de0148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 -- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1de015c640 msgr2=0x7f1de0148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1de015c640 0x7f1de0148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 -- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de0070f80 msgr2=0x7f1de0148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de0070f80 0x7f1de0148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 -- 172.21.15.77:0/1453137977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00094a0 con 0x7f1de015ca10 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de6ccd700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8001ba0 tx=0x7f1dd800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd800aac0 con 0x7f1de015ca10 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd800ac20 con 0x7f1de015ca10 2023-07-04T18:51:27.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de00c9090 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80070d0 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.111+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de00c9650 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.112+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de014ab00 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.113+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1dd8007230 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.113+0000 7f1dd6ffd700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 0x7f1dc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.113+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1dd8089310 con 0x7f1de015ca10 2023-07-04T18:51:27.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.113+0000 7f1de64cc700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 0x7f1dc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.114+0000 7f1de64cc700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 0x7f1dc406d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00090b0 tx=0x7f1dd0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.118+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1de014ab00 con 0x7f1de015ca10 2023-07-04T18:51:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.362+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1de014ab00 con 0x7f1de015ca10 2023-07-04T18:51:27.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.365+0000 7f1dd6ffd700 1 -- 172.21.15.77:0/1453137977 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170842 (secure 0 0 0) 0x7f1de014ab00 con 0x7f1de015ca10 2023-07-04T18:51:27.368 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 msgr2=0x7f1dc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 0x7f1dc406d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00090b0 tx=0x7f1dd0008030 comp rx=0 tx=0).stop 2023-07-04T18:51:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 msgr2=0x7f1de015b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8001ba0 tx=0x7f1dd800cf50 comp rx=0 tx=0).stop 2023-07-04T18:51:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 shutdown_connections 2023-07-04T18:51:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1dc406ace0 0x7f1dc406d190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1de015ca10 0x7f1de015b070 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de0070f80 0x7f1de0148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 --2- 172.21.15.77:0/1453137977 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1de015c640 0x7f1de0148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 >> 172.21.15.77:0/1453137977 conn(0x7f1de006bba0 msgr2=0x7f1de00cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 shutdown_connections 2023-07-04T18:51:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.368+0000 7f1de8730700 1 -- 172.21.15.77:0/1453137977 wait complete. 2023-07-04T18:51:27.381 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3872286367},{"type":"v1","addr":"172.21.15.82:6833","nonce":3872286367}]},"active_addr":"172.21.15.82:6833/3872286367","active_change":"2023-07-04T18:51:03.164932+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3637586579}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2731256261}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2743785258}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":2516995798}]}]} 2023-07-04T18:51:27.386 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-07-04T18:51:27.386 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-07-04T18:51:27.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:27.435 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T18:51:27.433+0000 7f161cf83700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-04T18:51:27.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.507+0000 7f6a7f1c3700 1 Processor -- start 2023-07-04T18:51:27.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.507+0000 7f6a7f1c3700 1 -- start start 2023-07-04T18:51:27.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7f1c3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a78158c90 0x7f6a780ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7f1c3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a780ce400 0x7f6a780ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7f1c3700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a78071ad0 con 0x7f6a78070f60 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7f1c3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a780cb9f0 con 0x7f6a78158c90 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7f1c3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780c9fd0 con 0x7f6a780ce400 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a78158c90 0x7f6a780ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6a78158c90 0x7f6a780ca3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43950/0 (socket says 172.21.15.77:43950) 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 -- 172.21.15.77:0/4180049516 learned_addr learned my addr 172.21.15.77:0/4180049516 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 -- 172.21.15.77:0/4180049516 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a780ce400 msgr2=0x7f6a780ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/4180049516 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a780ce400 0x7f6a780ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 -- 172.21.15.77:0/4180049516 >> v1:172.21.15.82:6789/0 conn(0x7f6a78070f60 legacy=0x7f6a780713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 -- 172.21.15.77:0/4180049516 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78071d50 con 0x7f6a78158c90 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/4180049516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78158c90 0x7f6a780ca3f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0098f0 tx=0x7f6a6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1028b60f811afcc server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a777fe700 1 -- 172.21.15.77:0/4180049516 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c004230 con 0x7f6a78158c90 2023-07-04T18:51:27.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a777fe700 1 -- 172.21.15.77:0/4180049516 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c004390 con 0x7f6a78158c90 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.508+0000 7f6a777fe700 1 -- 172.21.15.77:0/4180049516 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0036a0 con 0x7f6a78158c90 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/4180049516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78158c90 msgr2=0x7f6a780ca3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/4180049516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78158c90 0x7f6a780ca3f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c0098f0 tx=0x7f6a6c009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/4180049516 shutdown_connections 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/4180049516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a78158c90 0x7f6a780ca3f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/4180049516 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a780ce400 0x7f6a780ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/4180049516 >> 172.21.15.77:0/4180049516 conn(0x7f6a780cf220 msgr2=0x7f6a780cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:27.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/4180049516 shutdown_connections 2023-07-04T18:51:27.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.509+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/4180049516 wait complete. 2023-07-04T18:51:27.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.510+0000 7f6a7f1c3700 1 Processor -- start 2023-07-04T18:51:27.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.510+0000 7f6a7f1c3700 1 -- start start 2023-07-04T18:51:27.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 0x7f6a781f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a78158c90 0x7f6a78165150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78163760 con 0x7f6a780ce400 2023-07-04T18:51:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78162b50 con 0x7f6a78158c90 2023-07-04T18:51:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7f1c3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a78162e30 con 0x7f6a78070f60 2023-07-04T18:51:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43958/0 (socket says 172.21.15.77:43958) 2023-07-04T18:51:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 -- 172.21.15.77:0/2135227684 learned_addr learned my addr 172.21.15.77:0/2135227684 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7d760700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a78158c90 0x7f6a78165150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 0x7f6a781f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 -- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 msgr2=0x7f6a781f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 0x7f6a781f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 -- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a78158c90 msgr2=0x7f6a78165150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a78158c90 0x7f6a78165150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 -- 172.21.15.77:0/2135227684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c009550 con 0x7f6a780ce400 2023-07-04T18:51:27.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 0x7f6a781f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.511+0000 7f6a77fff700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b4a0 tx=0x7f6a6400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.512+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64011830 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.512+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64011990 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.512+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a78163110 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.512+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64011d40 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.512+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a781f54b0 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.513+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a780c7610 con 0x7f6a780ce400 2023-07-04T18:51:27.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.513+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a6401a400 con 0x7f6a780ce400 2023-07-04T18:51:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.514+0000 7f6a757fa700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 0x7f6a5c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.514+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6a64089440 con 0x7f6a780ce400 2023-07-04T18:51:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.514+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 0x7f6a5c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.514+0000 7f6a7cf5f700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 0x7f6a5c06d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009130 tx=0x7f6a6c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.518+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a64054500 con 0x7f6a780ce400 2023-07-04T18:51:27.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.708+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a7804e210 con 0x7f6a780ce400 2023-07-04T18:51:27.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.709+0000 7f6a757fa700 1 -- 172.21.15.77:0/2135227684 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f6a64057b20 con 0x7f6a780ce400 2023-07-04T18:51:27.710 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:27.710 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:27.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 msgr2=0x7f6a5c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 0x7f6a5c06d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009130 tx=0x7f6a6c008030 comp rx=0 tx=0).stop 2023-07-04T18:51:27.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 msgr2=0x7f6a78164c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400b4a0 tx=0x7f6a6400b7b0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 shutdown_connections 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6a5c06ace0 0x7f6a5c06d190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a78158c90 0x7f6a78165150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6a78070f60 0x7f6a781f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 --2- 172.21.15.77:0/2135227684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a780ce400 0x7f6a78164c10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 >> 172.21.15.77:0/2135227684 conn(0x7f6a780cf220 msgr2=0x7f6a780cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:27.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 shutdown_connections 2023-07-04T18:51:27.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.711+0000 7f6a7f1c3700 1 -- 172.21.15.77:0/2135227684 wait complete. 2023-07-04T18:51:27.724 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-07-04T18:51:27.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:27.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.844+0000 7f154d6a6700 1 Processor -- start 2023-07-04T18:51:27.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.844+0000 7f154d6a6700 1 -- start start 2023-07-04T18:51:27.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f154d6a6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f154815ca10 0x7f1548070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f154d6a6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15480cf830 con 0x7f154815c640 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f154d6a6700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15480711d0 con 0x7f1548070db0 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f154d6a6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15480717f0 con 0x7f154815ca10 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f1546ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f154815c640 0x7f15480cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51102/0 (socket says 172.21.15.77:51102) 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f15467fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f154815ca10 0x7f1548070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.845+0000 7f1546ffd700 1 -- 172.21.15.77:0/1719715865 learned_addr learned my addr 172.21.15.77:0/1719715865 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f15467fc700 1 -- 172.21.15.77:0/1719715865 >> v1:172.21.15.82:6789/0 conn(0x7f1548070db0 legacy=0x7f15480cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f15467fc700 1 -- 172.21.15.77:0/1719715865 >> v1:172.21.15.77:6789/0 conn(0x7f154815c640 legacy=0x7f15480cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f1545ffb700 1 -- 172.21.15.77:0/1719715865 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216705039 0 0) 0x7f15480cf830 con 0x7f154815c640 2023-07-04T18:51:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f15467fc700 1 -- 172.21.15.77:0/1719715865 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480cd290 con 0x7f154815ca10 2023-07-04T18:51:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f15467fc700 1 --2- 172.21.15.77:0/1719715865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f1548070870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1538009840 tx=0x7f1538009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59251916c09167b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f1545ffb700 1 -- 172.21.15.77:0/1719715865 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538011830 con 0x7f154815ca10 2023-07-04T18:51:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f1545ffb700 1 -- 172.21.15.77:0/1719715865 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1538011990 con 0x7f154815ca10 2023-07-04T18:51:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.846+0000 7f1545ffb700 1 -- 172.21.15.77:0/1719715865 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1538011d20 con 0x7f154815ca10 2023-07-04T18:51:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 -- 172.21.15.77:0/1719715865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 msgr2=0x7f1548070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1719715865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f1548070870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1538009840 tx=0x7f1538009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 -- 172.21.15.77:0/1719715865 shutdown_connections 2023-07-04T18:51:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1719715865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f1548070870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 -- 172.21.15.77:0/1719715865 >> 172.21.15.77:0/1719715865 conn(0x7f154806bba0 msgr2=0x7f15480b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:27.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 -- 172.21.15.77:0/1719715865 shutdown_connections 2023-07-04T18:51:27.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.847+0000 7f154d6a6700 1 -- 172.21.15.77:0/1719715865 wait complete. 2023-07-04T18:51:27.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.848+0000 7f154d6a6700 1 Processor -- start 2023-07-04T18:51:27.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.848+0000 7f154d6a6700 1 -- start start 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 0x7f15480afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f154815c640 0x7f1548156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1548158290 con 0x7f154815ca10 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15480b03f0 con 0x7f154815c640 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f154d6a6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1548157680 con 0x7f1548070db0 2023-07-04T18:51:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f15477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f15477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43972/0 (socket says 172.21.15.77:43972) 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.849+0000 7f15477fe700 1 -- 172.21.15.77:0/1457627622 learned_addr learned my addr 172.21.15.77:0/1457627622 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15467fc700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f154815c640 0x7f1548156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f1546ffd700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 0x7f15480afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 -- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 msgr2=0x7f15480afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 0x7f15480afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 -- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f154815c640 msgr2=0x7f1548156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f154815c640 0x7f1548156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 -- 172.21.15.77:0/1457627622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15380094a0 con 0x7f154815ca10 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f1546ffd700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 0x7f15480afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f15477fe700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f154000cb50 tx=0x7f154000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540003c00 con 0x7f154815ca10 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1540003d60 con 0x7f154815ca10 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1548157960 con 0x7f154815ca10 2023-07-04T18:51:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154000ab70 con 0x7f154815ca10 2023-07-04T18:51:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.850+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1548157f20 con 0x7f154815ca10 2023-07-04T18:51:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.851+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1548158400 con 0x7f154815ca10 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.851+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f154000acd0 con 0x7f154815ca10 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.852+0000 7f152f7fe700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 0x7f153406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.852+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f15400894c0 con 0x7f154815ca10 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.852+0000 7f1546ffd700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 0x7f153406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.852+0000 7f1546ffd700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 0x7f153406d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f153c000f60 tx=0x7f153c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:27.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:27.856+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1540054570 con 0x7f154815ca10 2023-07-04T18:51:28.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.044+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f154804e210 con 0x7f154815ca10 2023-07-04T18:51:28.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.045+0000 7f152f7fe700 1 -- 172.21.15.77:0/1457627622 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f1540057b90 con 0x7f154815ca10 2023-07-04T18:51:28.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:28.046 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:28.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 msgr2=0x7f153406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 0x7f153406d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f153c000f60 tx=0x7f153c009040 comp rx=0 tx=0).stop 2023-07-04T18:51:28.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 msgr2=0x7f15480cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f154000cb50 tx=0x7f154000cf10 comp rx=0 tx=0).stop 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 shutdown_connections 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f153406ace0 0x7f153406d190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f154815ca10 0x7f15480cb690 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1548070db0 0x7f15480afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 --2- 172.21.15.77:0/1457627622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f154815c640 0x7f1548156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.050 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 >> 172.21.15.77:0/1457627622 conn(0x7f154806bba0 msgr2=0x7f15481e6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.050 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 shutdown_connections 2023-07-04T18:51:28.050 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.047+0000 7f154d6a6700 1 -- 172.21.15.77:0/1457627622 wait complete. 2023-07-04T18:51:28.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-07-04T18:51:28.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-07-04T18:51:28.063 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-07-04T18:51:28.063 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-07-04T18:51:28.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-07-04T18:51:28.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-07-04T18:51:28.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-07-04T18:51:28.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-07-04T18:51:28.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.274+0000 7f2401403700 1 Processor -- start 2023-07-04T18:51:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.274+0000 7fd114b6e700 1 Processor -- start 2023-07-04T18:51:28.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.274+0000 7fd114b6e700 1 -- start start 2023-07-04T18:51:28.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.275+0000 7f905010a700 1 Processor -- start 2023-07-04T18:51:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7fd114b6e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd110158da0 0x7fd11006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7fd114b6e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11015d430 con 0x7fd11006bfc0 2023-07-04T18:51:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7fd114b6e700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd110159250 con 0x7fd1100cda00 2023-07-04T18:51:28.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7fd114b6e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd110159ed0 con 0x7fd110158da0 2023-07-04T18:51:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.275+0000 7f2401403700 1 -- start start 2023-07-04T18:51:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.275+0000 7f905010a700 1 -- start start 2023-07-04T18:51:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f905010a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f904815be90 0x7f90480cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f905010a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480ad700 0x7f90480adb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f905010a700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904815c260 con 0x7f90480cdb40 2023-07-04T18:51:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f905010a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90480ca690 con 0x7f904815be90 2023-07-04T18:51:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f905010a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90480ca970 con 0x7f90480ad700 2023-07-04T18:51:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f2401403700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 0x7f23fc0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f2401403700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f23fc15ea40 0x7f23fc0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f2401403700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0cf100 con 0x7f23fc0cebf0 2023-07-04T18:51:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f2401403700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23fc15df60 con 0x7f23fc15ea40 2023-07-04T18:51:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.276+0000 7f2401403700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc15e240 con 0x7f23fc15a430 2023-07-04T18:51:28.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f036cee5700 1 Processor -- start 2023-07-04T18:51:28.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f23faffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 0x7f23fc0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f036cee5700 1 -- start start 2023-07-04T18:51:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f2f15a4e700 1 Processor -- start 2023-07-04T18:51:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f23faffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 0x7f23fc0ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33832/0 (socket says 172.21.15.77:33832) 2023-07-04T18:51:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f23faffd700 1 -- 172.21.15.77:0/2280330426 learned_addr learned my addr 172.21.15.77:0/2280330426 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036cee5700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 0x7f036806c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036cee5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03680ca7d0 0x7f03680cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036cee5700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036815a7e0 con 0x7f036815df30 2023-07-04T18:51:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036cee5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680cb570 con 0x7f03680ca7d0 2023-07-04T18:51:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036cee5700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03680cb290 con 0x7f036815a410 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.277+0000 7f2f15a4e700 1 -- start start 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f2f15a4e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2f100ad700 0x7f2f100adb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f2f15a4e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f100cdb40 0x7f2f100cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f2f15a4e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100ce500 con 0x7f2f1015be90 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f2f15a4e700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100ccb90 con 0x7f2f100ad700 2023-07-04T18:51:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f2f15a4e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1015c260 con 0x7f2f100cdb40 2023-07-04T18:51:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036659c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 0x7f036806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036659c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 0x7f036806c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49804/0 (socket says 172.21.15.77:49804) 2023-07-04T18:51:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f036659c700 1 -- 172.21.15.77:0/2638272105 learned_addr learned my addr 172.21.15.77:0/2638272105 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.278+0000 7f23fb7fe700 1 --2- 172.21.15.77:0/2280330426 >> v2:172.21.15.77:3300/0 conn(0x7f23fc15ea40 0x7f23fc0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7fd10dd9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd11006bfc0 0x7fd11015c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51114/0 (socket says 172.21.15.77:51114) 2023-07-04T18:51:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7fd10dd9b700 1 -- 172.21.15.77:0/3462615200 learned_addr learned my addr 172.21.15.77:0/3462615200 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7fd10e59c700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd110158da0 0x7fd11006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f0365d9b700 1 --2- 172.21.15.77:0/2638272105 >> v2:172.21.15.77:3300/0 conn(0x7f03680ca7d0 0x7f03680cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f2f0e7fc700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2f100ad700 0x7f2f100adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f904e6a7700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f90480cdb40 0x7f90480cdfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59042/0 (socket says 172.21.15.77:59042) 2023-07-04T18:51:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f904e6a7700 1 -- 172.21.15.77:0/3928678992 learned_addr learned my addr 172.21.15.77:0/3928678992 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f2f0e7fc700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2f100ad700 0x7f2f100adb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49806/0 (socket says 172.21.15.77:49806) 2023-07-04T18:51:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/2006126775 learned_addr learned my addr 172.21.15.77:0/2006126775 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23fb7fe700 1 -- 172.21.15.77:0/2280330426 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 msgr2=0x7f23fc0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23fb7fe700 1 --2- 172.21.15.77:0/2280330426 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 0x7f23fc0ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23fb7fe700 1 -- 172.21.15.77:0/2280330426 >> v1:172.21.15.82:6789/0 conn(0x7f23fc0cebf0 legacy=0x7f23fc0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23fb7fe700 1 -- 172.21.15.77:0/2280330426 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc15e520 con 0x7f23fc15ea40 2023-07-04T18:51:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767625583 0 0) 0x7fd11015d430 con 0x7fd11006bfc0 2023-07-04T18:51:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f80035b0 con 0x7fd11006bfc0 2023-07-04T18:51:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23fb7fe700 1 --2- 172.21.15.77:0/2280330426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc15ea40 0x7f23fc0cfaf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009d90 tx=0x7f23f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0abff4ab55e4128 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712297579 0 0) 0x7f904815c260 con 0x7f90480cdb40 2023-07-04T18:51:28.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90340035b0 con 0x7f90480cdb40 2023-07-04T18:51:28.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f0365d9b700 1 -- 172.21.15.77:0/2638272105 >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 msgr2=0x7f036806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f0365d9b700 1 --2- 172.21.15.77:0/2638272105 >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 0x7f036806c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f0365d9b700 1 -- 172.21.15.77:0/2638272105 >> v1:172.21.15.82:6789/0 conn(0x7f036815df30 legacy=0x7f036815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f0365d9b700 1 -- 172.21.15.77:0/2638272105 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368069ee0 con 0x7f03680ca7d0 2023-07-04T18:51:28.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783345524 0 0) 0x7fd0f80035b0 con 0x7fd11006bfc0 2023-07-04T18:51:28.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.279+0000 7f23f9ffb700 1 -- 172.21.15.77:0/2280330426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f400e070 con 0x7f23fc15ea40 2023-07-04T18:51:28.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f23f9ffb700 1 -- 172.21.15.77:0/2280330426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f400bad0 con 0x7f23fc15ea40 2023-07-04T18:51:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd110158da0 msgr2=0x7fd11006ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f23f9ffb700 1 -- 172.21.15.77:0/2280330426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f400be80 con 0x7f23fc15ea40 2023-07-04T18:51:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd110158da0 0x7fd11006ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 >> v1:172.21.15.82:6789/0 conn(0x7fd1100cda00 legacy=0x7fd11015cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11015f970 con 0x7fd11006bfc0 2023-07-04T18:51:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fd104001670 con 0x7fd11006bfc0 2023-07-04T18:51:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f100cdb40 msgr2=0x7f2f100cdfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f2f0e7fc700 1 --2- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f100cdb40 0x7f2f100cdfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/2006126775 >> v1:172.21.15.77:6789/0 conn(0x7f2f1015be90 legacy=0x7f2f100cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/2006126775 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100ca690 con 0x7f2f100ad700 2023-07-04T18:51:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 >> v1:172.21.15.77:6789/0 conn(0x7fd11006bfc0 legacy=0x7fd11015c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0f8003a30 0x7fd0f8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0f8004410 0x7fd0f80048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f8005080 0x7fd0f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd110159ed0 con 0x7fd0f8005080 2023-07-04T18:51:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f800bf80 con 0x7fd0f8003a30 2023-07-04T18:51:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f800c140 con 0x7fd0f8004410 2023-07-04T18:51:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1011701565 0 0) 0x7f90340035b0 con 0x7f90480cdb40 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480ad700 msgr2=0x7f90480adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480ad700 0x7f90480adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 >> v2:172.21.15.77:3300/0 conn(0x7f904815be90 msgr2=0x7f90480cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 --2- 172.21.15.77:0/3928678992 >> v2:172.21.15.77:3300/0 conn(0x7f904815be90 0x7f90480cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480cac50 con 0x7f90480cdb40 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f90400019f0 con 0x7f90480cdb40 2023-07-04T18:51:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 >> v1:172.21.15.82:6789/0 conn(0x7f90480cdb40 legacy=0x7f90480cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034003a00 0x7f9034003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9034004850 0x7f9034004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9034005320 0x7f903400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90480ca690 con 0x7f9034004850 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90480ca970 con 0x7f9034003a00 2023-07-04T18:51:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903400c1b0 con 0x7f9034005320 2023-07-04T18:51:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0f8004410 0x7fd0f80048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f0365d9b700 1 --2- 172.21.15.77:0/2638272105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03680ca7d0 0x7f03680cac70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f035c009840 tx=0x7f035c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5183ec72124b10f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904d6a5700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9034004850 0x7f9034004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904e6a7700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9034005320 0x7f903400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10ed9d700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f8005080 0x7fd0f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10e59c700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0f8003a30 0x7fd0f8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f036559a700 1 -- 172.21.15.77:0/2638272105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c004230 con 0x7f03680ca7d0 2023-07-04T18:51:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f036559a700 1 -- 172.21.15.77:0/2638272105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c004390 con 0x7f03680ca7d0 2023-07-04T18:51:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f036559a700 1 -- 172.21.15.77:0/2638272105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0036a0 con 0x7f03680ca7d0 2023-07-04T18:51:28.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7f904dea6700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034003a00 0x7f9034003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 -- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0f8003a30 msgr2=0x7fd0f8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0f8003a30 0x7fd0f8003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 -- 172.21.15.77:0/3462615200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f8005080 msgr2=0x7fd0f800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0f8005080 0x7fd0f800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.280+0000 7fd10dd9b700 1 -- 172.21.15.77:0/3462615200 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11015f970 con 0x7fd0f8004410 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 -- 172.21.15.77:0/2280330426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc15ea40 msgr2=0x7f23fc0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 --2- 172.21.15.77:0/2280330426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc15ea40 0x7f23fc0cfaf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009d90 tx=0x7f23f400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:28.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0f8004410 0x7fd0f80048d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd10400c340 tx=0x7fd0f8005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59053ceb69b1807c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 -- 172.21.15.77:0/2280330426 shutdown_connections 2023-07-04T18:51:28.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 --2- 172.21.15.77:0/2280330426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc15ea40 0x7f23fc0cfaf0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 --2- 172.21.15.77:0/2280330426 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc15a430 0x7f23fc0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 -- 172.21.15.77:0/2280330426 >> 172.21.15.77:0/2280330426 conn(0x7f23fc06d7c0 msgr2=0x7f23fc06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2f0e7fc700 1 --2- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100ad700 0x7f2f100adb50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f000092e0 tx=0x7f2f000095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977acee321605d8a server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 -- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9034005320 msgr2=0x7f903400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9034005320 0x7f903400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 -- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034003a00 msgr2=0x7f9034003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034003a00 0x7f9034003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 -- 172.21.15.77:0/3928678992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480cac50 con 0x7f9034004850 2023-07-04T18:51:28.306 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10400cea0 con 0x7fd0f8004410 2023-07-04T18:51:28.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 -- 172.21.15.77:0/2280330426 shutdown_connections 2023-07-04T18:51:28.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd104005960 con 0x7fd0f8004410 2023-07-04T18:51:28.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7fd10d59a700 1 -- 172.21.15.77:0/3462615200 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104005d10 con 0x7fd0f8004410 2023-07-04T18:51:28.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f2401403700 1 -- 172.21.15.77:0/2280330426 wait complete. 2023-07-04T18:51:28.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904d6a5700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9034004850 0x7f9034004cc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90480c9a60 tx=0x7f9034005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cbcbe2b8407abd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9038011620 con 0x7f9034004850 2023-07-04T18:51:28.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9038011780 con 0x7f9034004850 2023-07-04T18:51:28.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.281+0000 7f904cea4700 1 -- 172.21.15.77:0/3928678992 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9038011b30 con 0x7f9034004850 2023-07-04T18:51:28.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 -- 172.21.15.77:0/2638272105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03680ca7d0 msgr2=0x7f03680cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 --2- 172.21.15.77:0/2638272105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03680ca7d0 0x7f03680cac70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f035c009840 tx=0x7f035c009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:28.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f2f0dffb700 1 -- 172.21.15.77:0/2006126775 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00004230 con 0x7f2f100ad700 2023-07-04T18:51:28.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 -- 172.21.15.77:0/2638272105 shutdown_connections 2023-07-04T18:51:28.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 --2- 172.21.15.77:0/2638272105 >> v2:172.21.15.82:3300/0 conn(0x7f036815a410 0x7f036806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 --2- 172.21.15.77:0/2638272105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03680ca7d0 0x7f03680cac70 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f035c009840 tx=0x7f035c009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:28.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f036cee5700 1 -- 172.21.15.77:0/2638272105 >> 172.21.15.77:0/2638272105 conn(0x7f03680cf220 msgr2=0x7f03680cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f2f0dffb700 1 -- 172.21.15.77:0/2006126775 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f00004390 con 0x7f2f100ad700 2023-07-04T18:51:28.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7fd114b6e700 1 -- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0f8004410 msgr2=0x7fd0f80048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7fd114b6e700 1 --2- 172.21.15.77:0/3462615200 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0f8004410 0x7fd0f80048d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd10400c340 tx=0x7fd0f8005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f905010a700 1 -- 172.21.15.77:0/3928678992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9034004850 msgr2=0x7f9034004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f905010a700 1 --2- 172.21.15.77:0/3928678992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9034004850 0x7f9034004cc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f90480c9a60 tx=0x7f9034005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f904e6a7700 1 -- 172.21.15.77:0/3928678992 reap_dead start 2023-07-04T18:51:28.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f905010a700 1 -- 172.21.15.77:0/3928678992 shutdown_connections 2023-07-04T18:51:28.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f905010a700 1 -- 172.21.15.77:0/3928678992 >> 172.21.15.77:0/3928678992 conn(0x7f904806aad0 msgr2=0x7f90480b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2401403700 1 Processor -- start 2023-07-04T18:51:28.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7fd10ed9d700 1 -- 172.21.15.77:0/3462615200 reap_dead start 2023-07-04T18:51:28.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7fd114b6e700 1 -- 172.21.15.77:0/3462615200 shutdown_connections 2023-07-04T18:51:28.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7fd114b6e700 1 -- 172.21.15.77:0/3462615200 >> 172.21.15.77:0/3462615200 conn(0x7fd11006aaa0 msgr2=0x7fd1100b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.282+0000 7f2f0dffb700 1 -- 172.21.15.77:0/2006126775 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000c770 con 0x7f2f100ad700 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 -- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100ad700 msgr2=0x7f2f100adb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100ad700 0x7f2f100adb50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2f000092e0 tx=0x7f2f000095f0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 -- 172.21.15.77:0/2006126775 shutdown_connections 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f100cdb40 0x7f2f100cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/2006126775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100ad700 0x7f2f100adb50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 -- 172.21.15.77:0/2006126775 >> 172.21.15.77:0/2006126775 conn(0x7f2f1006aad0 msgr2=0x7f2f100b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 -- 172.21.15.77:0/2006126775 shutdown_connections 2023-07-04T18:51:28.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f917249a700 1 Processor -- start 2023-07-04T18:51:28.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f434872b700 1 Processor -- start 2023-07-04T18:51:28.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f434872b700 1 -- start start 2023-07-04T18:51:28.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2f15a4e700 1 -- 172.21.15.77:0/2006126775 wait complete. 2023-07-04T18:51:28.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2f15a4e700 1 Processor -- start 2023-07-04T18:51:28.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f434872b700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f43400cb530 0x7f43400ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f434872b700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400adf80 con 0x7f43400cb900 2023-07-04T18:51:28.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f434872b700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400cc710 con 0x7f43400acdf0 2023-07-04T18:51:28.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f434872b700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400ad370 con 0x7f43400cb530 2023-07-04T18:51:28.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f2401403700 1 -- start start 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc0cebf0 0x7f23fc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23fc15ea40 0x7f23fc159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc0c6650 0x7f23fc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0c4180 con 0x7f23fc0c6650 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0c5640 con 0x7f23fc15ea40 2023-07-04T18:51:28.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2401403700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc0c5890 con 0x7f23fc0cebf0 2023-07-04T18:51:28.321 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f036cee5700 1 -- 172.21.15.77:0/2638272105 shutdown_connections 2023-07-04T18:51:28.321 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f23fb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc0c6650 0x7f23fc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.321 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f23fb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc0c6650 0x7f23fc0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44024/0 (socket says 172.21.15.77:44024) 2023-07-04T18:51:28.321 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f23fb7fe700 1 -- 172.21.15.77:0/4111974948 learned_addr learned my addr 172.21.15.77:0/4111974948 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.322 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f4345cc6700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f43400acdf0 0x7f43400c90b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59076/0 (socket says 172.21.15.77:59076) 2023-07-04T18:51:28.322 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f4345cc6700 1 -- 172.21.15.77:0/2805299536 learned_addr learned my addr 172.21.15.77:0/2805299536 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.322 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f43464c7700 1 --2- 172.21.15.77:0/2805299536 >> v2:172.21.15.82:3300/0 conn(0x7f43400cb530 0x7f43400ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.323 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f036cee5700 1 -- 172.21.15.77:0/2638272105 wait complete. 2023-07-04T18:51:28.323 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7fd114b6e700 1 -- 172.21.15.77:0/3462615200 shutdown_connections 2023-07-04T18:51:28.323 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.283+0000 7f917249a700 1 -- start start 2023-07-04T18:51:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f917249a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f916c160ab0 0x7f916c164f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f917249a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f916c0cd730 0x7f916c06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f917249a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c06b810 con 0x7f916c0ad700 2023-07-04T18:51:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f917249a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c06c1e0 con 0x7f916c160ab0 2023-07-04T18:51:28.324 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f917249a700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0ce1b0 con 0x7f916c0cd730 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43454c5700 1 -- 172.21.15.77:0/2805299536 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58743543 0 0) 0x7f43400adf80 con 0x7f43400cb900 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43454c5700 1 -- 172.21.15.77:0/2805299536 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432c0035b0 con 0x7f43400cb900 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43464c7700 1 -- 172.21.15.77:0/2805299536 >> v1:172.21.15.82:6789/0 conn(0x7f43400acdf0 legacy=0x7f43400c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43464c7700 1 -- 172.21.15.77:0/2805299536 >> v1:172.21.15.77:6789/0 conn(0x7f43400cb900 legacy=0x7f4340160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43464c7700 1 -- 172.21.15.77:0/2805299536 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400ad650 con 0x7f43400cb530 2023-07-04T18:51:28.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f916bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f916c160ab0 0x7f916c164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f916bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f916c160ab0 0x7f916c164f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44032/0 (socket says 172.21.15.77:44032) 2023-07-04T18:51:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f916bfff700 1 -- 172.21.15.77:0/3197978061 learned_addr learned my addr 172.21.15.77:0/3197978061 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.284+0000 7f2f15a4e700 1 -- start start 2023-07-04T18:51:28.326 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43464c7700 1 --2- 172.21.15.77:0/2805299536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400cb530 0x7f43400ac8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4338009d90 tx=0x7f433800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=155aeb08328da184 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f905010a700 1 -- 172.21.15.77:0/3928678992 shutdown_connections 2023-07-04T18:51:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43454c5700 1 -- 172.21.15.77:0/2805299536 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433800e070 con 0x7f43400cb530 2023-07-04T18:51:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43454c5700 1 -- 172.21.15.77:0/2805299536 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433800bb20 con 0x7f43400cb530 2023-07-04T18:51:28.327 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f43454c5700 1 -- 172.21.15.77:0/2805299536 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338003a20 con 0x7f43400cb530 2023-07-04T18:51:28.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f100ad700 0x7f2f100b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 0x7f2f101591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f1015be90 0x7f2f100b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f101596e0 con 0x7f2f100ad700 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100b19d0 con 0x7f2f100cdb40 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f15a4e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100b2280 con 0x7f2f1015be90 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 0x7f2f101591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 0x7f2f101591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49844/0 (socket says 172.21.15.77:49844) 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/1881838602 learned_addr learned my addr 172.21.15.77:0/1881838602 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f434872b700 1 -- 172.21.15.77:0/2805299536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400cb530 msgr2=0x7f43400ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f434872b700 1 --2- 172.21.15.77:0/2805299536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400cb530 0x7f43400ac8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4338009d90 tx=0x7f433800b330 comp rx=0 tx=0).stop 2023-07-04T18:51:28.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f036cee5700 1 Processor -- start 2023-07-04T18:51:28.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f434872b700 1 -- 172.21.15.77:0/2805299536 shutdown_connections 2023-07-04T18:51:28.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f434872b700 1 --2- 172.21.15.77:0/2805299536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400cb530 0x7f43400ac8b0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4338009d90 tx=0x7f433800b330 comp rx=0 tx=0).stop 2023-07-04T18:51:28.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f434872b700 1 -- 172.21.15.77:0/2805299536 >> 172.21.15.77:0/2805299536 conn(0x7f434006aad0 msgr2=0x7f43400b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f1015be90 msgr2=0x7f2f100b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2f1015be90 0x7f2f100b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/1881838602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f100ad700 msgr2=0x7f2f100b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f100ad700 0x7f2f100b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f2f0e7fc700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00009de0 con 0x7f2f100cdb40 2023-07-04T18:51:28.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f2f0e7fc700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 0x7f2f101591a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7fd114b6e700 1 -- 172.21.15.77:0/3462615200 wait complete. 2023-07-04T18:51:28.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f434872b700 1 -- 172.21.15.77:0/2805299536 shutdown_connections 2023-07-04T18:51:28.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7fd114b6e700 1 Processor -- start 2023-07-04T18:51:28.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f434872b700 1 -- 172.21.15.77:0/2805299536 wait complete. 2023-07-04T18:51:28.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f916b7fe700 1 --2- 172.21.15.77:0/3197978061 >> v2:172.21.15.82:3300/0 conn(0x7f916c0cd730 0x7f916c06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650797322 0 0) 0x7f916c06b810 con 0x7f916c0ad700 2023-07-04T18:51:28.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91540035b0 con 0x7f916c0ad700 2023-07-04T18:51:28.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924379102 0 0) 0x7f91540035b0 con 0x7f916c0ad700 2023-07-04T18:51:28.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 >> v2:172.21.15.82:3300/0 conn(0x7f916c0cd730 msgr2=0x7f916c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 --2- 172.21.15.77:0/3197978061 >> v2:172.21.15.82:3300/0 conn(0x7f916c0cd730 0x7f916c06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 >> v2:172.21.15.77:3300/0 conn(0x7f916c160ab0 msgr2=0x7f916c164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 --2- 172.21.15.77:0/3197978061 >> v2:172.21.15.77:3300/0 conn(0x7f916c160ab0 0x7f916c164f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0cded0 con 0x7f916c0ad700 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f91640019f0 con 0x7f916c0ad700 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 >> v1:172.21.15.77:6789/0 conn(0x7f916c0ad700 legacy=0x7f916c0c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9154003a50 0x7f9154003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9154004850 0x7f9154004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f915400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c06c1e0 con 0x7f9154004850 2023-07-04T18:51:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0ce1b0 con 0x7f9154003a50 2023-07-04T18:51:28.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915400c210 con 0x7f9154005250 2023-07-04T18:51:28.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f23faffd700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc0cebf0 0x7f23fc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f23fa7fc700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23fc15ea40 0x7f23fc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.285+0000 7f23faffd700 1 -- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23fc15ea40 msgr2=0x7f23fc159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23faffd700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23fc15ea40 0x7f23fc159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23faffd700 1 -- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc0c6650 msgr2=0x7f23fc0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23faffd700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23fc0c6650 0x7f23fc0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23faffd700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4009a70 con 0x7f23fc0cebf0 2023-07-04T18:51:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23faffd700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc0cebf0 0x7f23fc0cae80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00e810 tx=0x7f23ec00eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec00c070 con 0x7f23fc0cebf0 2023-07-04T18:51:28.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec013610 con 0x7f23fc0cebf0 2023-07-04T18:51:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.286+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec0139c0 con 0x7f23fc0cebf0 2023-07-04T18:51:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f915400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f905010a700 1 -- 172.21.15.77:0/3928678992 wait complete. 2023-07-04T18:51:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 -- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9154003a50 msgr2=0x7f9154003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9154003a50 0x7f9154003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 -- 172.21.15.77:0/3197978061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9154004850 msgr2=0x7f9154004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9154004850 0x7f9154004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f9170a37700 1 -- 172.21.15.77:0/3197978061 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540035b0 con 0x7f9154005250 2023-07-04T18:51:28.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f9170a37700 1 -- 172.21.15.77:0/3197978061 reap_dead start 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f036cee5700 1 -- start start 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f036cee5700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036815a410 0x7f0368154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f036cee5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 0x7f0368155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.287+0000 7f036cee5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f036815d050 0x7f0368156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f036cee5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680aec80 con 0x7f036815d050 2023-07-04T18:51:28.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f036cee5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03680b2a60 con 0x7f036815df30 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f036cee5700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03680b3020 con 0x7f036815a410 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 0x7f0368155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 0x7f0368155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49856/0 (socket says 172.21.15.77:49856) 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 -- 172.21.15.77:0/3489736079 learned_addr learned my addr 172.21.15.77:0/3489736079 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 -- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036815a410 msgr2=0x7f0368154800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036815a410 0x7f0368154800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 -- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f036815d050 msgr2=0x7f0368156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f036815d050 0x7f0368156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0094a0 con 0x7f036815df30 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0365d9b700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 0x7f0368155c10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f035c004000 tx=0x7f035c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0045d0 con 0x7f036815df30 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c01e070 con 0x7f036815df30 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036815d4c0 con 0x7f036815df30 2023-07-04T18:51:28.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.288+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c011850 con 0x7f036815df30 2023-07-04T18:51:28.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f434872b700 1 Processor -- start 2023-07-04T18:51:28.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03680b2cb0 con 0x7f036815df30 2023-07-04T18:51:28.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f434872b700 1 -- start start 2023-07-04T18:51:28.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f036806a4a0 con 0x7f036815df30 2023-07-04T18:51:28.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f905010a700 1 Processor -- start 2023-07-04T18:51:28.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.289+0000 7f905010a700 1 -- start start 2023-07-04T18:51:28.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f9170a37700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f915400ba10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9154005830 tx=0x7f9154005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=408852d369cb32c8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400acdf0 0x7f43400c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 0x7f43400c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43401591d0 0x7f43400c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43401584a0 con 0x7f43401591d0 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434015b9b0 con 0x7f43400acdf0 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f434872b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43401586f0 con 0x7f43400cb900 2023-07-04T18:51:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7fd114b6e700 1 -- start start 2023-07-04T18:51:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0001e070 con 0x7f2f100cdb40 2023-07-04T18:51:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0000c050 con 0x7f2f100cdb40 2023-07-04T18:51:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f000038a0 con 0x7f2f100cdb40 2023-07-04T18:51:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164016070 con 0x7f9154005250 2023-07-04T18:51:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f035c0119b0 con 0x7f036815df30 2023-07-04T18:51:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f4345cc6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 0x7f43400c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f43464c7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400acdf0 0x7f43400c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f4345cc6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 0x7f43400c23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33904/0 (socket says 172.21.15.77:33904) 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f4345cc6700 1 -- 172.21.15.77:0/470782719 learned_addr learned my addr 172.21.15.77:0/470782719 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f4346cc8700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43401591d0 0x7f43400c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 -- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400acdf0 msgr2=0x7f43400c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400acdf0 0x7f43400c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 -- 172.21.15.77:0/470782719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43401591d0 msgr2=0x7f43400c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43401591d0 0x7f43400c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338009a70 con 0x7f43400cb900 2023-07-04T18:51:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0356ffd700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f035006f270 0x7f0350071720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f43464c7700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43400acdf0 0x7f43400c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:28.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f035c08aa90 con 0x7f036815df30 2023-07-04T18:51:28.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4345cc6700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 0x7f43400c23e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f433c00b550 tx=0x7f433c00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100b2560 con 0x7f2f100cdb40 2023-07-04T18:51:28.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100b2b20 con 0x7f2f100cdb40 2023-07-04T18:51:28.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1100c66e0 0x7fd1100c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1100c76f0 0x7fd1100af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd11014fff0 0x7fd1100af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f036659c700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f035006f270 0x7f0350071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1101ff200 con 0x7fd1100c66e0 2023-07-04T18:51:28.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1101504f0 con 0x7fd1100c76f0 2023-07-04T18:51:28.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd114b6e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1101507d0 con 0x7fd11014fff0 2023-07-04T18:51:28.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0356ffd700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] conn(0x7f0350077aa0 0x7f0350079eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f035007a560 con 0x7f0350077aa0 2023-07-04T18:51:28.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f035c08aee0 con 0x7f036815df30 2023-07-04T18:51:28.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9164012440 con 0x7f9154005250 2023-07-04T18:51:28.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f916affd700 1 -- 172.21.15.77:0/3197978061 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640127f0 con 0x7f9154005250 2023-07-04T18:51:28.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f917249a700 1 -- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 msgr2=0x7f915400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f917249a700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f915400ba10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9154005830 tx=0x7f9154005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd10e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1100c66e0 0x7fd1100c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd10e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1100c66e0 0x7fd1100c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44064/0 (socket says 172.21.15.77:44064) 2023-07-04T18:51:28.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd10dd9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1100c76f0 0x7fd1100af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7fd10e59c700 1 -- 172.21.15.77:0/1561401220 learned_addr learned my addr 172.21.15.77:0/1561401220 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7fd10ed9d700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd11014fff0 0x7fd1100af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f0366d9d700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] conn(0x7f0350077aa0 0x7f0350079eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc0c5b70 con 0x7f23fc0cebf0 2023-07-04T18:51:28.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc0c6130 con 0x7f23fc0cebf0 2023-07-04T18:51:28.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c011830 con 0x7f43400cb900 2023-07-04T18:51:28.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c0119b0 con 0x7f43400cb900 2023-07-04T18:51:28.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c011d60 con 0x7f43400cb900 2023-07-04T18:51:28.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43401589d0 con 0x7f43400cb900 2023-07-04T18:51:28.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43400c5110 con 0x7f43400cb900 2023-07-04T18:51:28.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f100cac50 con 0x7f2f100cdb40 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034004850 0x7f90480c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90480c7d20 0x7f9048150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 0x7f90480c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90480c4850 con 0x7f90480c7d20 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90480c6880 con 0x7f9034004850 2023-07-04T18:51:28.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f905010a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9048151320 con 0x7f90480c3ea0 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f904e6a7700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 0x7f90480c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f904e6a7700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 0x7f90480c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33910/0 (socket says 172.21.15.77:33910) 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.290+0000 7f904e6a7700 1 -- 172.21.15.77:0/1111741558 learned_addr learned my addr 172.21.15.77:0/1111741558 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 -- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034004850 msgr2=0x7f90480c6340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9034004850 0x7f90480c6340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 -- 172.21.15.77:0/1111741558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90480c7d20 msgr2=0x7f9048150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90480c7d20 0x7f9048150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480c4c70 con 0x7f90480c3ea0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f904e6a7700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 0x7f90480c4310 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90480c9fb0 tx=0x7f904000ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904000d8a0 con 0x7f90480c3ea0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904000da00 con 0x7f90480c3ea0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.291+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90480c6b00 con 0x7f90480c3ea0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904000dd70 con 0x7f90480c3ea0 2023-07-04T18:51:28.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90480c70c0 con 0x7f90480c3ea0 2023-07-04T18:51:28.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f2becf59700 1 Processor -- start 2023-07-04T18:51:28.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f0366d9d700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] conn(0x7f0350077aa0 0x7f0350079eb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f917249a700 1 -- 172.21.15.77:0/3197978061 shutdown_connections 2023-07-04T18:51:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f917249a700 1 --2- 172.21.15.77:0/3197978061 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f915400ba10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f917249a700 1 -- 172.21.15.77:0/3197978061 >> 172.21.15.77:0/3197978061 conn(0x7f916c06aad0 msgr2=0x7f916c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f43400adc10 con 0x7f43400cb900 2023-07-04T18:51:28.365 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f036659c700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f035006f270 0x7f0350071720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0358009a60 tx=0x7f03580092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.292+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f90480caf30 con 0x7f90480c3ea0 2023-07-04T18:51:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 -- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd11014fff0 msgr2=0x7fd1100af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd11014fff0 0x7fd1100af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 -- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1100c66e0 msgr2=0x7fd1100c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1100c66e0 0x7fd1100c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd110150ab0 con 0x7fd1100c76f0 2023-07-04T18:51:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd10dd9b700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1100c76f0 0x7fd1100af270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd104014000 tx=0x7fd104001a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9040015430 con 0x7f90480c3ea0 2023-07-04T18:51:28.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7f903e7fc700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f902c06ad30 0x7f902c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f90400899e0 con 0x7f90480c3ea0 2023-07-04T18:51:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f903e7fc700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] conn(0x7f902c0735e0 0x7f902c0759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f902c0760a0 con 0x7f902c0735e0 2023-07-04T18:51:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9040089e80 con 0x7f90480c3ea0 2023-07-04T18:51:28.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f904d6a5700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] conn(0x7f902c0735e0 0x7f902c0759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f2becf59700 1 -- start start 2023-07-04T18:51:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2becf59700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2becf59700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2be8071350 0x7f2be806c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2becf59700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be80ce640 con 0x7f2be806cba0 2023-07-04T18:51:28.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2becf59700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80cd690 con 0x7f2be8071350 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2becf59700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80ccf80 con 0x7f2be806d730 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:33924/0 (socket says 172.21.15.77:33924) 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 -- 172.21.15.77:0/1716055277 learned_addr learned my addr 172.21.15.77:0/1716055277 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23fc06ef20 con 0x7f23fc0cebf0 2023-07-04T18:51:28.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f23ec013b20 con 0x7f23fc0cebf0 2023-07-04T18:51:28.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== osd.2 v2:172.21.15.77:6824/3740385611 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f035007a560 con 0x7f0350077aa0 2023-07-04T18:51:28.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f917249a700 1 -- 172.21.15.77:0/3197978061 shutdown_connections 2023-07-04T18:51:28.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.293+0000 7f917249a700 1 -- 172.21.15.77:0/3197978061 wait complete. 2023-07-04T18:51:28.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f917249a700 1 Processor -- start 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f917249a700 1 -- start start 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104001c80 con 0x7fd1100c76f0 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1040072e0 con 0x7fd1100c76f0 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd110150d90 con 0x7fd1100c76f0 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1100c6ab0 con 0x7fd1100c76f0 2023-07-04T18:51:28.372 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104007690 con 0x7fd1100c76f0 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0f4000fc0 con 0x7fd1100c76f0 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f23e37fe700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f23e806ace0 0x7f23e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f23ec088e90 con 0x7f23fc0cebf0 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23fa7fc700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f23e806ace0 0x7f23e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23e37fe700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] conn(0x7f23e8073490 0x7f23e80758a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23e8075f50 con 0x7f23e8073490 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23fb7fe700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] conn(0x7f23e8073490 0x7f23e80758a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23ec053f40 con 0x7f23fc0cebf0 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.294+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f433c01a400 con 0x7f43400cb900 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4332ffd700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f432406adb0 0x7f432406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f433c089ab0 con 0x7f43400cb900 2023-07-04T18:51:28.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f43464c7700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f432406adb0 0x7f432406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4332ffd700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] conn(0x7f4324073660 0x7f4324075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4324076120 con 0x7f4324073660 2023-07-04T18:51:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f433c089e60 con 0x7f43400cb900 2023-07-04T18:51:28.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f4346cc8700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] conn(0x7f4324073660 0x7f4324075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f43464c7700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f432406adb0 0x7f432406d260 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf950 tx=0x7f4338011000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2f00003c10 con 0x7f2f100cdb40 2023-07-04T18:51:28.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f904dea6700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f902c06ad30 0x7f902c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f904dea6700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f902c06ad30 0x7f902c06d1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f904d6a5700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] conn(0x7f902c0735e0 0x7f902c0759f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f916c145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916c0c7060 0x7f916c146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 0x7f916c070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0c1ee0 con 0x7f916c0c7060 2023-07-04T18:51:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b4600 con 0x7f916c0b3510 2023-07-04T18:51:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f917249a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b3d60 con 0x7f9154005250 2023-07-04T18:51:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 -- 172.21.15.77:0/1716055277 >> v2:172.21.15.82:3300/0 conn(0x7f2be8071350 msgr2=0x7f2be806c660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 --2- 172.21.15.77:0/1716055277 >> v2:172.21.15.82:3300/0 conn(0x7f2be8071350 0x7f2be806c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 -- 172.21.15.77:0/1716055277 >> v1:172.21.15.82:6789/0 conn(0x7f2be806cba0 legacy=0x7f2be8069e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f23fb7fe700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] conn(0x7f23e8073490 0x7f23e80758a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.295+0000 7f2be659c700 1 -- 172.21.15.77:0/1716055277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80ce8e0 con 0x7f2be806d730 2023-07-04T18:51:28.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.296+0000 7f2be659c700 1 --2- 172.21.15.77:0/1716055277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0098f0 tx=0x7f2bdc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be327bba4feb03b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f23fa7fc700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f23e806ace0 0x7f23e806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23fc15dec0 tx=0x7f23f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2ef7fff700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2efc06adb0 0x7f2efc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f4346cc8700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] conn(0x7f4324073660 0x7f4324075a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2be559a700 1 -- 172.21.15.77:0/1716055277 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004230 con 0x7f2be806d730 2023-07-04T18:51:28.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2be559a700 1 -- 172.21.15.77:0/1716055277 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc004390 con 0x7f2be806d730 2023-07-04T18:51:28.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2be559a700 1 -- 172.21.15.77:0/1716055277 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0036a0 con 0x7f2be806d730 2023-07-04T18:51:28.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2f0008a830 con 0x7f2f100cdb40 2023-07-04T18:51:28.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 0x7f916c070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 0x7f916c070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49898/0 (socket says 172.21.15.77:49898) 2023-07-04T18:51:28.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 -- 172.21.15.77:0/908482813 learned_addr learned my addr 172.21.15.77:0/908482813 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2f0effd700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2efc06adb0 0x7f2efc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f916bfff700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f916c145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2ef7fff700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] conn(0x7f2efc073660 0x7f2efc075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2efc076120 con 0x7f2efc073660 2023-07-04T18:51:28.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f0008ce50 con 0x7f2f100cdb40 2023-07-04T18:51:28.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== osd.7 v2:172.21.15.82:6824/3741415560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f902c0760a0 con 0x7f902c0735e0 2023-07-04T18:51:28.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 -- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 msgr2=0x7f916c145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9154005250 0x7f916c145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 -- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916c0c7060 msgr2=0x7f916c146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f916c0c7060 0x7f916c146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f9170a37700 1 -- 172.21.15.77:0/908482813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0b4040 con 0x7f916c0b3510 2023-07-04T18:51:28.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7f2f0f7fe700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] conn(0x7f2efc073660 0x7f2efc075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.297+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd10401d420 con 0x7fd1100c76f0 2023-07-04T18:51:28.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2f0effd700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2efc06adb0 0x7f2efc06d260 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800abb0 tx=0x7f2f08009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 -- 172.21.15.77:0/1716055277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 msgr2=0x7f2be8070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 --2- 172.21.15.77:0/1716055277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0098f0 tx=0x7f2bdc009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:28.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== osd.4 v2:172.21.15.82:6800/1338358939 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4324076120 con 0x7f4324073660 2023-07-04T18:51:28.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2f0f7fe700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] conn(0x7f2efc073660 0x7f2efc075a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 -- 172.21.15.77:0/1716055277 shutdown_connections 2023-07-04T18:51:28.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 --2- 172.21.15.77:0/1716055277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be806d730 0x7f2be8070ca0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 --2- 172.21.15.77:0/1716055277 >> v2:172.21.15.82:3300/0 conn(0x7f2be8071350 0x7f2be806c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 -- 172.21.15.77:0/1716055277 >> 172.21.15.77:0/1716055277 conn(0x7f2be806bba0 msgr2=0x7f2be80b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 -- 172.21.15.77:0/1716055277 shutdown_connections 2023-07-04T18:51:28.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2becf59700 1 -- 172.21.15.77:0/1716055277 wait complete. 2023-07-04T18:51:28.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd0feffd700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd0f006adb0 0x7fd0f006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd10408a150 con 0x7fd1100c76f0 2023-07-04T18:51:28.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd0feffd700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] conn(0x7fd0f00735e0 0x7fd0f00759f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0f00760a0 con 0x7fd0f00735e0 2023-07-04T18:51:28.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd10e59c700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd0f006adb0 0x7fd0f006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7fd10ed9d700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] conn(0x7fd0f00735e0 0x7fd0f00759f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f9170a37700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 0x7f916c070160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9164002230 tx=0x7f9164001cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.298+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== osd.5 v2:172.21.15.82:6816/2297621363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2efc076120 con 0x7f2efc073660 2023-07-04T18:51:28.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd104055270 con 0x7fd1100c76f0 2023-07-04T18:51:28.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7fd10ed9d700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] conn(0x7fd0f00735e0 0x7fd0f00759f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.300+0000 7f2becf59700 1 Processor -- start 2023-07-04T18:51:28.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.300+0000 7f2becf59700 1 -- start start 2023-07-04T18:51:28.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164016070 con 0x7f916c0b3510 2023-07-04T18:51:28.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9164017040 con 0x7f916c0b3510 2023-07-04T18:51:28.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9164005740 con 0x7f916c0b3510 2023-07-04T18:51:28.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c0b4320 con 0x7f916c0b3510 2023-07-04T18:51:28.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.299+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c146b80 con 0x7f916c0b3510 2023-07-04T18:51:28.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== osd.3 v2:172.21.15.77:6816/1589412867 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f23e8075f50 con 0x7f23e8073490 2023-07-04T18:51:28.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 0x7f2be8161cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be806d730 0x7f2be8142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be8071350 0x7f2be8157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be8157c60 con 0x7f2be806d730 2023-07-04T18:51:28.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be8161210 con 0x7f2be806cba0 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.301+0000 7f2becf59700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be81614f0 con 0x7f2be8071350 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 0x7f2be8161cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 0x7f2be8161cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49906/0 (socket says 172.21.15.77:49906) 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 -- 172.21.15.77:0/4087944467 learned_addr learned my addr 172.21.15.77:0/4087944467 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7fd10e59c700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd0f006adb0 0x7fd0f006d260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1000046e0 tx=0x7fd100008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== osd.1 v2:172.21.15.77:6800/1069379231 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd0f00760a0 con 0x7fd0f00735e0 2023-07-04T18:51:28.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be5d9b700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be806d730 0x7f2be8142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 -- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be8071350 msgr2=0x7f2be8157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2be8071350 0x7f2be8157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 -- 172.21.15.77:0/4087944467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be806d730 msgr2=0x7f2be8142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2be806d730 0x7f2be8142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc009550 con 0x7f2be806cba0 2023-07-04T18:51:28.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f91640125b0 con 0x7f916c0b3510 2023-07-04T18:51:28.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f9168ff9700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f915806ad30 0x7f915806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f916408e170 con 0x7f916c0b3510 2023-07-04T18:51:28.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f916bfff700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f915806ad30 0x7f915806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.302+0000 7f2be659c700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 0x7f2be8161cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009c40 tx=0x7f2bdc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc00b660 con 0x7f2be806cba0 2023-07-04T18:51:28.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bdc00b7c0 con 0x7f2be806cba0 2023-07-04T18:51:28.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc00bb30 con 0x7f2be806cba0 2023-07-04T18:51:28.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f916bfff700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f915806ad30 0x7f915806d1e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f916c0af760 tx=0x7f915c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be81617d0 con 0x7f2be806cba0 2023-07-04T18:51:28.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.303+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be8157f40 con 0x7f2be806cba0 2023-07-04T18:51:28.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.304+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2be8161380 con 0x7f2be806cba0 2023-07-04T18:51:28.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.308+0000 7f917249a700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] conn(0x7f914c0015e0 0x7f914c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.308+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 --> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f914c006bc0 con 0x7f914c0015e0 2023-07-04T18:51:28.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.309+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2bdc022070 con 0x7f2be806cba0 2023-07-04T18:51:28.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.309+0000 7f916b7fe700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] conn(0x7f914c0015e0 0x7f914c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.309+0000 7f2bd6ffd700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2bc806ace0 0x7f2bc806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2bdc08b020 con 0x7f2be806cba0 2023-07-04T18:51:28.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2bd6ffd700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] conn(0x7f2bc8073490 0x7f2bc80758a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2bc8075f50 con 0x7f2bc8073490 2023-07-04T18:51:28.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f916b7fe700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] conn(0x7f914c0015e0 0x7f914c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bdc056280 con 0x7f2be806cba0 2023-07-04T18:51:28.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2be5d9b700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2bc806ace0 0x7f2bc806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.310+0000 7f2be6d9d700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] conn(0x7f2bc8073490 0x7f2bc80758a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.311+0000 7f2be5d9b700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2bc806ace0 0x7f2bc806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2be80c3350 tx=0x7f2bd80090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.311+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== osd.0 v2:172.21.15.77:6808/4272426713 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f914c006bc0 con 0x7f914c0015e0 2023-07-04T18:51:28.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.312+0000 7f2be6d9d700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] conn(0x7f2bc8073490 0x7f2bc80758a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.312+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== osd.6 v2:172.21.15.82:6808/2390446703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2bc8075f50 con 0x7f2bc8073490 2023-07-04T18:51:28.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.325+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 --> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f036806a1c0 con 0x7f0350077aa0 2023-07-04T18:51:28.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.325+0000 7f0356ffd700 1 -- 172.21.15.77:0/3489736079 <== osd.2 v2:172.21.15.77:6824/3740385611 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f036000a720 con 0x7f0350077aa0 2023-07-04T18:51:28.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] conn(0x7f0350077aa0 msgr2=0x7f0350079eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.77:6824/3740385611,v1:172.21.15.77:6825/3740385611] conn(0x7f0350077aa0 0x7f0350079eb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f035006f270 msgr2=0x7f0350071720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f035006f270 0x7f0350071720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0358009a60 tx=0x7f03580092c0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 msgr2=0x7f0368155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f036cee5700 1 --2- 172.21.15.77:0/3489736079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f036815df30 0x7f0368155c10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f035c004000 tx=0x7f035c004750 comp rx=0 tx=0).stop 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.327+0000 7f0366d9d700 1 -- 172.21.15.77:0/3489736079 reap_dead start 2023-07-04T18:51:28.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.329+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 shutdown_connections 2023-07-04T18:51:28.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.329+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 >> 172.21.15.77:0/3489736079 conn(0x7f03680cf220 msgr2=0x7f03680ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.329+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 --> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f904804e210 con 0x7f902c0735e0 2023-07-04T18:51:28.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.329+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 --> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f23fc0c6350 con 0x7f23e8073490 2023-07-04T18:51:28.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f903e7fc700 1 -- 172.21.15.77:0/1111741558 <== osd.7 v2:172.21.15.82:6824/3741415560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f904804e210 con 0x7f902c0735e0 2023-07-04T18:51:28.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f23e37fe700 1 -- 172.21.15.77:0/4111974948 <== osd.3 v2:172.21.15.77:6816/1589412867 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f23fc0c6350 con 0x7f23e8073490 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] conn(0x7f902c0735e0 msgr2=0x7f902c0759f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6824/3741415560,v1:172.21.15.82:6825/3741415560] conn(0x7f902c0735e0 0x7f902c0759f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f902c06ad30 msgr2=0x7f902c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f902c06ad30 0x7f902c06d1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9044006fd0 tx=0x7f9044008040 comp rx=0 tx=0).stop 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 msgr2=0x7f90480c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 --2- 172.21.15.77:0/1111741558 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90480c3ea0 0x7f90480c4310 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90480c9fb0 tx=0x7f904000ff80 comp rx=0 tx=0).stop 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f904e6a7700 1 -- 172.21.15.77:0/1111741558 reap_dead start 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 shutdown_connections 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 >> 172.21.15.77:0/1111741558 conn(0x7f904806aad0 msgr2=0x7f90480cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 shutdown_connections 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.330+0000 7f905010a700 1 -- 172.21.15.77:0/1111741558 wait complete. 2023-07-04T18:51:28.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.331+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 --> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f100ca970 con 0x7f2efc073660 2023-07-04T18:51:28.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.331+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 shutdown_connections 2023-07-04T18:51:28.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.331+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 --> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f43400ad930 con 0x7f4324073660 2023-07-04T18:51:28.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.331+0000 7f036cee5700 1 -- 172.21.15.77:0/3489736079 wait complete. 2023-07-04T18:51:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.332+0000 7f4332ffd700 1 -- 172.21.15.77:0/470782719 <== osd.4 v2:172.21.15.82:6800/1338358939 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f43340014f0 con 0x7f4324073660 2023-07-04T18:51:28.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.332+0000 7f2ef7fff700 1 -- 172.21.15.77:0/1881838602 <== osd.5 v2:172.21.15.82:6816/2297621363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f04009280 con 0x7f2efc073660 2023-07-04T18:51:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.332+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] conn(0x7f2efc073660 msgr2=0x7f2efc075a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.332+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6816/2297621363,v1:172.21.15.82:6817/2297621363] conn(0x7f2efc073660 0x7f2efc075a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2efc06adb0 msgr2=0x7f2efc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2efc06adb0 0x7f2efc06d260 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800abb0 tx=0x7f2f08009250 comp rx=0 tx=0).stop 2023-07-04T18:51:28.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 msgr2=0x7f2f101591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2f15a4e700 1 --2- 172.21.15.77:0/1881838602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2f100cdb40 0x7f2f101591a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00004750 comp rx=0 tx=0).stop 2023-07-04T18:51:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2f0f7fe700 1 -- 172.21.15.77:0/1881838602 reap_dead start 2023-07-04T18:51:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] conn(0x7f4324073660 msgr2=0x7f4324075a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6800/1338358939,v1:172.21.15.82:6801/1338358939] conn(0x7f4324073660 0x7f4324075a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f432406adb0 msgr2=0x7f432406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f432406adb0 0x7f432406d260 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf950 tx=0x7f4338011000 comp rx=0 tx=0).stop 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 msgr2=0x7f43400c23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 --2- 172.21.15.77:0/470782719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f43400cb900 0x7f43400c23e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f433c00b550 tx=0x7f433c00b860 comp rx=0 tx=0).stop 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f4346cc8700 1 -- 172.21.15.77:0/470782719 reap_dead start 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 shutdown_connections 2023-07-04T18:51:28.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 >> 172.21.15.77:0/470782719 conn(0x7f434006aad0 msgr2=0x7f43400ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] conn(0x7f23e8073490 msgr2=0x7f23e80758a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.77:6816/1589412867,v1:172.21.15.77:6817/1589412867] conn(0x7f23e8073490 0x7f23e80758a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 shutdown_connections 2023-07-04T18:51:28.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f434872b700 1 -- 172.21.15.77:0/470782719 wait complete. 2023-07-04T18:51:28.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f23e806ace0 msgr2=0x7f23e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f23e806ace0 0x7f23e806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23fc15dec0 tx=0x7f23f0009040 comp rx=0 tx=0).stop 2023-07-04T18:51:28.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc0cebf0 msgr2=0x7f23fc0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 --2- 172.21.15.77:0/4111974948 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f23fc0cebf0 0x7f23fc0cae80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00e810 tx=0x7f23ec00eb20 comp rx=0 tx=0).stop 2023-07-04T18:51:28.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f23fb7fe700 1 -- 172.21.15.77:0/4111974948 reap_dead start 2023-07-04T18:51:28.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 shutdown_connections 2023-07-04T18:51:28.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 >> 172.21.15.77:0/4111974948 conn(0x7f23fc06d7c0 msgr2=0x7f23fc15a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 shutdown_connections 2023-07-04T18:51:28.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.333+0000 7f2401403700 1 -- 172.21.15.77:0/4111974948 wait complete. 2023-07-04T18:51:28.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.335+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 shutdown_connections 2023-07-04T18:51:28.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.335+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 >> 172.21.15.77:0/1881838602 conn(0x7f2f1006aad0 msgr2=0x7f2f1015af10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.335+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 shutdown_connections 2023-07-04T18:51:28.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.335+0000 7f2f15a4e700 1 -- 172.21.15.77:0/1881838602 wait complete. 2023-07-04T18:51:28.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.338+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 --> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f914c005c70 con 0x7f914c0015e0 2023-07-04T18:51:28.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.338+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 --> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0f4002d40 con 0x7fd0f00735e0 2023-07-04T18:51:28.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.338+0000 7f9168ff9700 1 -- 172.21.15.77:0/908482813 <== osd.0 v2:172.21.15.77:6808/4272426713 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f914c005c70 con 0x7f914c0015e0 2023-07-04T18:51:28.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.342+0000 7fd0feffd700 1 -- 172.21.15.77:0/1561401220 <== osd.1 v2:172.21.15.77:6800/1069379231 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0f4002d40 con 0x7fd0f00735e0 2023-07-04T18:51:28.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.344+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 --> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2be80cd260 con 0x7f2bc8073490 2023-07-04T18:51:28.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.344+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] conn(0x7f914c0015e0 msgr2=0x7f914c003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.344+0000 7f917249a700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.77:6808/4272426713,v1:172.21.15.77:6809/4272426713] conn(0x7f914c0015e0 0x7f914c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.344+0000 7f2bd6ffd700 1 -- 172.21.15.77:0/4087944467 <== osd.6 v2:172.21.15.82:6808/2390446703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2be000a700 con 0x7f2bc8073490 2023-07-04T18:51:28.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f915806ad30 msgr2=0x7f915806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f915806ad30 0x7f915806d1e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f916c0af760 tx=0x7f915c008040 comp rx=0 tx=0).stop 2023-07-04T18:51:28.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 msgr2=0x7f916c070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 --2- 172.21.15.77:0/908482813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f916c0b3510 0x7f916c070160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9164002230 tx=0x7f9164001cb0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f9170a37700 1 -- 172.21.15.77:0/908482813 reap_dead start 2023-07-04T18:51:28.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 shutdown_connections 2023-07-04T18:51:28.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 >> 172.21.15.77:0/908482813 conn(0x7f916c06aad0 msgr2=0x7f916c071070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 shutdown_connections 2023-07-04T18:51:28.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.345+0000 7f917249a700 1 -- 172.21.15.77:0/908482813 wait complete. 2023-07-04T18:51:28.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] conn(0x7fd0f00735e0 msgr2=0x7fd0f00759f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7fd114b6e700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.77:6800/1069379231,v1:172.21.15.77:6801/1069379231] conn(0x7fd0f00735e0 0x7fd0f00759f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] conn(0x7f2bc8073490 msgr2=0x7f2bc80758a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6808/2390446703,v1:172.21.15.82:6809/2390446703] conn(0x7f2bc8073490 0x7f2bc80758a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2bc806ace0 msgr2=0x7f2bc806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2bc806ace0 0x7f2bc806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2be80c3350 tx=0x7f2bd80090e0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 msgr2=0x7f2be8161cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2becf59700 1 --2- 172.21.15.77:0/4087944467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2be806cba0 0x7f2be8161cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009c40 tx=0x7f2bdc004750 comp rx=0 tx=0).stop 2023-07-04T18:51:28.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.347+0000 7f2be6d9d700 1 -- 172.21.15.77:0/4087944467 reap_dead start 2023-07-04T18:51:28.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.350+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 shutdown_connections 2023-07-04T18:51:28.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.350+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 >> 172.21.15.77:0/4087944467 conn(0x7f2be806bba0 msgr2=0x7f2be8069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.350+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 shutdown_connections 2023-07-04T18:51:28.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.350+0000 7f2becf59700 1 -- 172.21.15.77:0/4087944467 wait complete. 2023-07-04T18:51:28.427 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:28.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-07-04T18:51:28.428 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:28.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-07-04T18:51:28.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.354+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd0f006adb0 msgr2=0x7fd0f006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.354+0000 7fd114b6e700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd0f006adb0 0x7fd0f006d260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd1000046e0 tx=0x7fd100008040 comp rx=0 tx=0).stop 2023-07-04T18:51:28.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.354+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1100c76f0 msgr2=0x7fd1100af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.354+0000 7fd114b6e700 1 --2- 172.21.15.77:0/1561401220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1100c76f0 0x7fd1100af270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd104014000 tx=0x7fd104001a00 comp rx=0 tx=0).stop 2023-07-04T18:51:28.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.354+0000 7fd10ed9d700 1 -- 172.21.15.77:0/1561401220 reap_dead start 2023-07-04T18:51:28.431 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:28.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-04T18:51:28.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.356+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 shutdown_connections 2023-07-04T18:51:28.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.356+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 >> 172.21.15.77:0/1561401220 conn(0x7fd11006aaa0 msgr2=0x7fd110070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.432 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:28.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-04T18:51:28.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.356+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 shutdown_connections 2023-07-04T18:51:28.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.356+0000 7fd114b6e700 1 -- 172.21.15.77:0/1561401220 wait complete. 2023-07-04T18:51:28.434 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:28.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-07-04T18:51:28.435 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:28.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-04T18:51:28.436 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:28.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-07-04T18:51:28.438 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:28.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-04T18:51:28.677 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.675+0000 7f65b4348700 1 Processor -- start 2023-07-04T18:51:28.677 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.675+0000 7f65b4348700 1 -- start start 2023-07-04T18:51:28.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b4348700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac06b120 0x7f65ac0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b4348700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b4348700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac16dcf0 con 0x7f65ac16cdd0 2023-07-04T18:51:28.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b4348700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ac0ce0f0 con 0x7f65ac0cdc70 2023-07-04T18:51:28.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b4348700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac06b9c0 con 0x7f65ac06b120 2023-07-04T18:51:28.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.676+0000 7fe658922700 1 Processor -- start 2023-07-04T18:51:28.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f2a9bec7700 1 Processor -- start 2023-07-04T18:51:28.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b28e5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b28e5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44118/0 (socket says 172.21.15.77:44118) 2023-07-04T18:51:28.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b28e5700 1 -- 172.21.15.77:0/661125759 learned_addr learned my addr 172.21.15.77:0/661125759 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7f65b18e3700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac06b120 0x7f65ac0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.677+0000 7fe658922700 1 -- start start 2023-07-04T18:51:28.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f65b28e5700 1 -- 172.21.15.77:0/661125759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac06b120 msgr2=0x7f65ac0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f65b28e5700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac06b120 0x7f65ac0cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f65b28e5700 1 -- 172.21.15.77:0/661125759 >> v1:172.21.15.82:6789/0 conn(0x7f65ac16cdd0 legacy=0x7f65ac16d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f65b28e5700 1 -- 172.21.15.77:0/661125759 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac06bca0 con 0x7f65ac0cdc70 2023-07-04T18:51:28.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f65b28e5700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f659c0098f0 tx=0x7f659c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c3fe5b1fd1d95f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.678+0000 7f2a9bec7700 1 -- start start 2023-07-04T18:51:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f65b10e2700 1 -- 172.21.15.77:0/661125759 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c004230 con 0x7f65ac0cdc70 2023-07-04T18:51:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f65b10e2700 1 -- 172.21.15.77:0/661125759 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c004390 con 0x7f65ac0cdc70 2023-07-04T18:51:28.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f65b10e2700 1 -- 172.21.15.77:0/661125759 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c0036a0 con 0x7f65ac0cdc70 2023-07-04T18:51:28.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f0c0c09b700 1 Processor -- start 2023-07-04T18:51:28.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe658922700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe65415be90 0x7fe6540ad700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe658922700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6540adc40 0x7fe6540cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe658922700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65415c2e0 con 0x7fe6540b0b50 2023-07-04T18:51:28.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe658922700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6540ccb90 con 0x7fe6540adc40 2023-07-04T18:51:28.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe658922700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ca690 con 0x7fe65415be90 2023-07-04T18:51:28.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe65259c700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe6540b0b50 0x7fe6540cc480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59172/0 (socket says 172.21.15.77:59172) 2023-07-04T18:51:28.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7fe65259c700 1 -- 172.21.15.77:0/1676896003 learned_addr learned my addr 172.21.15.77:0/1676896003 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f0c0c09b700 1 -- start start 2023-07-04T18:51:28.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786442673 0 0) 0x7fe65415c2e0 con 0x7fe6540b0b50 2023-07-04T18:51:28.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63c0035b0 con 0x7fe6540b0b50 2023-07-04T18:51:28.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe652d9d700 1 --2- 172.21.15.77:0/1676896003 >> v2:172.21.15.77:3300/0 conn(0x7fe6540adc40 0x7fe6540cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f2a9bec7700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 0x7f2a9415be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f2a9bec7700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2a940cc480 0x7f2a940ad700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f2a9bec7700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940ca690 con 0x7f2a940cc8d0 2023-07-04T18:51:28.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f2a9bec7700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940ca970 con 0x7f2a940cc480 2023-07-04T18:51:28.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.679+0000 7f2a9bec7700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940cac50 con 0x7f2a940b0b50 2023-07-04T18:51:28.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f65b4348700 1 -- 172.21.15.77:0/661125759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 msgr2=0x7f65ac0ad4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f65b4348700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f659c0098f0 tx=0x7f659c009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:28.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f2a9a464700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2a940cc8d0 0x7f2a940cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51270/0 (socket says 172.21.15.77:51270) 2023-07-04T18:51:28.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f2a9a464700 1 -- 172.21.15.77:0/249384286 learned_addr learned my addr 172.21.15.77:0/249384286 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f0c0c09b700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0c040b0b50 0x7f0c0415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f0c0c09b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040cc8d0 0x7f0c040cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f0c0c09b700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040ca690 con 0x7f0c040cc480 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f0c0c09b700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c040ade10 con 0x7f0c040b0b50 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7f0c0c09b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040ca970 con 0x7f0c040cc8d0 2023-07-04T18:51:28.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0c040b0b50 0x7f0c0415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 -- 172.21.15.77:0/661125759 shutdown_connections 2023-07-04T18:51:28.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0ad4f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 --2- 172.21.15.77:0/661125759 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac06b120 0x7f65ac0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 -- 172.21.15.77:0/661125759 >> 172.21.15.77:0/661125759 conn(0x7f65ac06aad0 msgr2=0x7f65ac0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074987980 0 0) 0x7f2a940ca690 con 0x7f2a940cc8d0 2023-07-04T18:51:28.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a800035b0 con 0x7f2a940cc8d0 2023-07-04T18:51:28.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0c040b0b50 0x7f0c0415bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49942/0 (socket says 172.21.15.77:49942) 2023-07-04T18:51:28.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 -- 172.21.15.77:0/4268392258 learned_addr learned my addr 172.21.15.77:0/4268392258 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725541678 0 0) 0x7fe63c0035b0 con 0x7fe6540b0b50 2023-07-04T18:51:28.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe65415be90 msgr2=0x7fe6540ad700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe65415be90 0x7fe6540ad700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 >> v2:172.21.15.77:3300/0 conn(0x7fe6540adc40 msgr2=0x7fe6540cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 --2- 172.21.15.77:0/1676896003 >> v2:172.21.15.77:3300/0 conn(0x7fe6540adc40 0x7fe6540cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540ca970 con 0x7fe6540b0b50 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fe648001670 con 0x7fe6540b0b50 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 >> v1:172.21.15.82:6789/0 conn(0x7fe6540b0b50 legacy=0x7fe6540cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe63c003a00 0x7fe63c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe63c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe63c0050b0 0x7fe63c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ccb90 con 0x7fe63c0050b0 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ca690 con 0x7fe63c0043e0 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c00c030 con 0x7fe63c003a00 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.680+0000 7fe651d9b700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe63c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 -- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe63c003a00 msgr2=0x7fe63c003e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:28.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe63c003a00 0x7fe63c003e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 -- 172.21.15.77:0/1676896003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe63c0050b0 msgr2=0x7fe63c00ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe63c0050b0 0x7fe63c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 -- 172.21.15.77:0/1676896003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540ca970 con 0x7fe63c0043e0 2023-07-04T18:51:28.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7fe651d9b700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe63c0048b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6540c9a60 tx=0x7fe63c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cd39ebf1f6947ac server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 -- 172.21.15.77:0/661125759 shutdown_connections 2023-07-04T18:51:28.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f2a99462700 1 --2- 172.21.15.77:0/249384286 >> v2:172.21.15.82:3300/0 conn(0x7f2a940cc480 0x7f2a940ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f2a99c63700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 0x7f2a9415be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 -- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040cc8d0 msgr2=0x7f0c040cdb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 --2- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040cc8d0 0x7f0c040cdb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 -- 172.21.15.77:0/4268392258 >> v1:172.21.15.82:6789/0 conn(0x7f0c040cc480 legacy=0x7f0c040ad700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 -- 172.21.15.77:0/4268392258 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c040cac50 con 0x7f0c040b0b50 2023-07-04T18:51:28.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f0c09e37700 1 --2- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040b0b50 0x7f0c0415bc20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009e50 tx=0x7f0bfc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b776d6f6d9146b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f0c08e35700 1 -- 172.21.15.77:0/4268392258 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00e070 con 0x7f0c040b0b50 2023-07-04T18:51:28.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f0c08e35700 1 -- 172.21.15.77:0/4268392258 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc003a20 con 0x7f0c040b0b50 2023-07-04T18:51:28.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f0c08e35700 1 -- 172.21.15.77:0/4268392258 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc003dd0 con 0x7f0c040b0b50 2023-07-04T18:51:28.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827456559 0 0) 0x7f2a800035b0 con 0x7f2a940cc8d0 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 msgr2=0x7f2a9415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 0x7f2a9415be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 >> v2:172.21.15.82:3300/0 conn(0x7f2a940cc480 msgr2=0x7f2a940ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 --2- 172.21.15.77:0/249384286 >> v2:172.21.15.82:3300/0 conn(0x7f2a940cc480 0x7f2a940ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940caf30 con 0x7f2a940cc8d0 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f2a8c0019f0 con 0x7f2a940cc8d0 2023-07-04T18:51:28.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 >> v1:172.21.15.77:6789/0 conn(0x7f2a940cc8d0 legacy=0x7f2a940cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a80003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 0x7f2a80004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a80004d90 0x7f2a80005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940ca970 con 0x7f2a80004d90 2023-07-04T18:51:28.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940cac50 con 0x7f2a80003a00 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8000ba10 con 0x7f2a940b0b50 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a80003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 -- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 msgr2=0x7f2a80004850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a940b0b50 0x7f2a80004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 -- 172.21.15.77:0/249384286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a80004d90 msgr2=0x7f2a80005200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a80004d90 0x7f2a80005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 -- 172.21.15.77:0/249384286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940caf30 con 0x7f2a80003a00 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f2a99c63700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a80003e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009b90 tx=0x7f2a80005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f76d2a7e6274bad server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400e6c0 con 0x7f2a80003a00 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8400e820 con 0x7f2a80003a00 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f2a98c61700 1 -- 172.21.15.77:0/249384286 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400eb90 con 0x7f2a80003a00 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644011620 con 0x7fe63c0043e0 2023-07-04T18:51:28.702 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644011780 con 0x7fe63c0043e0 2023-07-04T18:51:28.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7fe65159a700 1 -- 172.21.15.77:0/1676896003 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644011b10 con 0x7fe63c0043e0 2023-07-04T18:51:28.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f0c0c09b700 1 -- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040b0b50 msgr2=0x7f0c0415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040b0b50 0x7f0c0415bc20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009e50 tx=0x7f0bfc00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:28.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.681+0000 7f65b4348700 1 -- 172.21.15.77:0/661125759 wait complete. 2023-07-04T18:51:28.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f65b4348700 1 Processor -- start 2023-07-04T18:51:28.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f65b4348700 1 -- start start 2023-07-04T18:51:28.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7fe658922700 1 -- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 msgr2=0x7fe63c0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7fe658922700 1 --2- 172.21.15.77:0/1676896003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe63c0048b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6540c9a60 tx=0x7fe63c005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7fe652d9d700 1 -- 172.21.15.77:0/1676896003 reap_dead start 2023-07-04T18:51:28.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7fe658922700 1 -- 172.21.15.77:0/1676896003 shutdown_connections 2023-07-04T18:51:28.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7fe658922700 1 -- 172.21.15.77:0/1676896003 >> 172.21.15.77:0/1676896003 conn(0x7fe65406aad0 msgr2=0x7fe6540b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7fe658922700 1 -- 172.21.15.77:0/1676896003 shutdown_connections 2023-07-04T18:51:28.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 -- 172.21.15.77:0/4268392258 shutdown_connections 2023-07-04T18:51:28.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040cc8d0 0x7f0c040cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/4268392258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040b0b50 0x7f0c0415bc20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 -- 172.21.15.77:0/4268392258 >> 172.21.15.77:0/4268392258 conn(0x7f0c0406aad0 msgr2=0x7f0c040b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7fe658922700 1 -- 172.21.15.77:0/1676896003 wait complete. 2023-07-04T18:51:28.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f2a9bec7700 1 -- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 msgr2=0x7f2a80003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/249384286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a80003e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009b90 tx=0x7f2a80005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f2a9a464700 1 -- 172.21.15.77:0/249384286 reap_dead start 2023-07-04T18:51:28.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac16cdd0 0x7f65ac0c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac179ae0 con 0x7f65ac0cdc70 2023-07-04T18:51:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac179dc0 con 0x7f65ac06b120 2023-07-04T18:51:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b4348700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac17a0a0 con 0x7f65ac16cdd0 2023-07-04T18:51:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f2a9bec7700 1 -- 172.21.15.77:0/249384286 shutdown_connections 2023-07-04T18:51:28.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f2a9bec7700 1 -- 172.21.15.77:0/249384286 >> 172.21.15.77:0/249384286 conn(0x7f2a9406aad0 msgr2=0x7f2a940b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 -- 172.21.15.77:0/4268392258 shutdown_connections 2023-07-04T18:51:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b20e4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f0c0c09b700 1 -- 172.21.15.77:0/4268392258 wait complete. 2023-07-04T18:51:28.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b20e4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49956/0 (socket says 172.21.15.77:49956) 2023-07-04T18:51:28.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f65b20e4700 1 -- 172.21.15.77:0/3499153335 learned_addr learned my addr 172.21.15.77:0/3499153335 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b18e3700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b28e5700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac16cdd0 0x7f65ac0c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b20e4700 1 -- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac16cdd0 msgr2=0x7f65ac0c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b20e4700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac16cdd0 0x7f65ac0c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b20e4700 1 -- 172.21.15.77:0/3499153335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 msgr2=0x7f65ac0b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b20e4700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.712 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f65b20e4700 1 -- 172.21.15.77:0/3499153335 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c009550 con 0x7f65ac06b120 2023-07-04T18:51:28.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f2a9bec7700 1 -- 172.21.15.77:0/249384286 shutdown_connections 2023-07-04T18:51:28.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f16a5471700 1 Processor -- start 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.682+0000 7f16a5471700 1 -- start start 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.683+0000 7f16a5471700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16a016d850 0x7f16a00cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f16a5471700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a006b6e0 con 0x7f16a016cdd0 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f16a5471700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a006b9c0 con 0x7f16a006b120 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f16a5471700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a006bca0 con 0x7f16a016d850 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f169effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16a016d850 0x7f16a00cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f169effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16a016d850 0x7f16a00cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44144/0 (socket says 172.21.15.77:44144) 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f169effd700 1 -- 172.21.15.77:0/1733184647 learned_addr learned my addr 172.21.15.77:0/1733184647 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.684+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717679775 0 0) 0x7f16a006b6e0 con 0x7f16a016cdd0 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16880035b0 con 0x7f16a016cdd0 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778135266 0 0) 0x7f16a006b9c0 con 0x7f16a006b120 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a006b6e0 con 0x7f16a006b120 2023-07-04T18:51:28.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f65b20e4700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f65a400cc50 tx=0x7f65a400cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.716 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4192594969 0 0) 0x7f16880035b0 con 0x7f16a016cdd0 2023-07-04T18:51:28.716 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 >> v2:172.21.15.77:3300/0 conn(0x7f16a016d850 msgr2=0x7f16a00cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 --2- 172.21.15.77:0/1733184647 >> v2:172.21.15.77:3300/0 conn(0x7f16a016d850 0x7f16a00cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 >> v1:172.21.15.82:6789/0 conn(0x7f16a006b120 legacy=0x7f16a00adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7fe658922700 1 Processor -- start 2023-07-04T18:51:28.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400aad0 con 0x7f65ac06b120 2023-07-04T18:51:28.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a400ac30 con 0x7f65ac06b120 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a006bf80 con 0x7f16a016cdd0 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f1694001670 con 0x7f16a016cdd0 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 >> v1:172.21.15.77:6789/0 conn(0x7f16a016cdd0 legacy=0x7f16a00ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1688004bf0 0x7f16880037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 0x7f1688003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004270 0x7f16880079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980015b0 con 0x7f1688004fc0 2023-07-04T18:51:28.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a006bca0 con 0x7f1688004bf0 2023-07-04T18:51:28.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16880081c0 con 0x7f1688004270 2023-07-04T18:51:28.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f0c0c09b700 1 Processor -- start 2023-07-04T18:51:28.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169effd700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1688004bf0 0x7f16880037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f0c0c09b700 1 -- start start 2023-07-04T18:51:28.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7f169e7fc700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 0x7f1688003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f6ecff7e700 1 Processor -- start 2023-07-04T18:51:28.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a4012420 con 0x7f65ac06b120 2023-07-04T18:51:28.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 -- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004270 msgr2=0x7f16880079f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:28.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004270 0x7f16880079f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 -- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1688004bf0 msgr2=0x7f16880037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1688004bf0 0x7f16880037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 -- 172.21.15.77:0/1733184647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a006bf80 con 0x7f1688004fc0 2023-07-04T18:51:28.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac17a380 con 0x7f65ac06b120 2023-07-04T18:51:28.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac20f6e0 con 0x7f65ac06b120 2023-07-04T18:51:28.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.685+0000 7f2a9bec7700 1 -- 172.21.15.77:0/249384286 wait complete. 2023-07-04T18:51:28.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f2a9bec7700 1 Processor -- start 2023-07-04T18:51:28.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169e7fc700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 0x7f1688003d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f169400c1b0 tx=0x7f1688005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b926918541a02c33 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.686+0000 7fe658922700 1 -- start start 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe6540c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe6540c4760 0x7fe6540c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540c38d0 con 0x7fe6540ce320 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654152e70 con 0x7fe63c0043e0 2023-07-04T18:51:28.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7fe658922700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540c4f60 con 0x7fe6540c4760 2023-07-04T18:51:28.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f169f7fe700 1 -- 172.21.15.77:0/1733184647 reap_dead start 2023-07-04T18:51:28.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040cc8d0 0x7f0c0414d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 0x7f0c04145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040c7030 con 0x7f0c0414dcc0 2023-07-04T18:51:28.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040c7310 con 0x7f0c040cc8d0 2023-07-04T18:51:28.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f0c0c09b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040c75f0 con 0x7f0c040b0b50 2023-07-04T18:51:28.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34016/0 (socket says 172.21.15.77:34016) 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 -- 172.21.15.77:0/57543037 learned_addr learned my addr 172.21.15.77:0/57543037 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09636700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040cc8d0 0x7f0c0414d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400ccd0 con 0x7f1688004fc0 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169400ce30 con 0x7f1688004fc0 2023-07-04T18:51:28.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f169dffb700 1 -- 172.21.15.77:0/1733184647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694012620 con 0x7f1688004fc0 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.687+0000 7f2a9bec7700 1 -- start start 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a940b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 0x7f2a94153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 0x7f2a940b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94154150 con 0x7f2a941fe830 2023-07-04T18:51:28.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940b2e40 con 0x7f2a80003a00 2023-07-04T18:51:28.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f2a9bec7700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a940b3120 con 0x7f2a94153cb0 2023-07-04T18:51:28.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44162/0 (socket says 172.21.15.77:44162) 2023-07-04T18:51:28.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 -- 172.21.15.77:0/789334337 learned_addr learned my addr 172.21.15.77:0/789334337 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 -- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe6540c4760 msgr2=0x7fe6540c2920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:28.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe6540c4760 0x7fe6540c2920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 -- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 msgr2=0x7fe6540c4220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:28.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe6540c4220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7fe651d9b700 1 -- 172.21.15.77:0/789334337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540c4c80 con 0x7fe6540ce320 2023-07-04T18:51:28.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f6ecff7e700 1 -- start start 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c0a638700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 0x7f0c04145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 -- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040cc8d0 msgr2=0x7f0c0414d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040cc8d0 0x7f0c0414d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 -- 172.21.15.77:0/57543037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 msgr2=0x7f0c04145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 0x7f0c04145ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c09e37700 1 -- 172.21.15.77:0/57543037 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc009b30 con 0x7f0c040b0b50 2023-07-04T18:51:28.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f0c0a638700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 0x7f0c04145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:28.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a9a464700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 0x7f2a940b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a9a464700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 0x7f2a940b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34026/0 (socket says 172.21.15.77:34026) 2023-07-04T18:51:28.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a9a464700 1 -- 172.21.15.77:0/1671931857 learned_addr learned my addr 172.21.15.77:0/1671931857 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 0x7f2a94153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe651d9b700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe644009da0 tx=0x7fe644004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0c09e37700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b370 tx=0x7f0bfc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f16a5471700 1 -- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 msgr2=0x7f1688003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f16a5471700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 0x7f1688003d30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f169400c1b0 tx=0x7f1688005880 comp rx=0 tx=0).stop 2023-07-04T18:51:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac17a8a0 con 0x7f65ac06b120 2023-07-04T18:51:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0042c0 con 0x7f0c040b0b50 2023-07-04T18:51:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc004420 con 0x7f0c040b0b50 2023-07-04T18:51:28.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0047d0 con 0x7f0c040b0b50 2023-07-04T18:51:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 -- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 msgr2=0x7f2a940b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 0x7f2a940b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 -- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 msgr2=0x7f2a940b3ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a940b3ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f2a99462700 1 -- 172.21.15.77:0/1671931857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940b3400 con 0x7f2a941fe830 2023-07-04T18:51:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f65a4012580 con 0x7f65ac06b120 2023-07-04T18:51:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f65a2ffd700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 0x7f659006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f65a4089b90 con 0x7f65ac06b120 2023-07-04T18:51:28.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f65b18e3700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 0x7f659006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f65b18e3700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 0x7f659006d1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0ad7f0 tx=0x7f65a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f16a5471700 1 -- 172.21.15.77:0/1733184647 shutdown_connections 2023-07-04T18:51:28.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f16a5471700 1 --2- 172.21.15.77:0/1733184647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1688004fc0 0x7f1688003d30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f16a5471700 1 -- 172.21.15.77:0/1733184647 >> 172.21.15.77:0/1733184647 conn(0x7f16a006aad0 msgr2=0x7f16a00b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f16a5471700 1 -- 172.21.15.77:0/1733184647 shutdown_connections 2023-07-04T18:51:28.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f16a5471700 1 -- 172.21.15.77:0/1733184647 wait complete. 2023-07-04T18:51:28.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f2a99462700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 0x7f2a94153770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406dfd0 tx=0x7f2a9000eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644004220 con 0x7fe6540ce320 2023-07-04T18:51:28.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6440043a0 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644006ce0 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe654152a90 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540c3a60 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6541528c0 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe644003740 con 0x7fe6540ce320 2023-07-04T18:51:28.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7fe642ffd700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 0x7fe63406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec8164f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ec80cd730 0x7f6ec806bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6ec80ad700 0x7f6ec80adb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec80ae0c0 con 0x7f6ec80ad700 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec806b810 con 0x7f6ec80cd730 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecff7e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec806c1e0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecd519700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ec80cd730 0x7f6ec806bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecd519700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6ec80cd730 0x7f6ec806bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:49994/0 (socket says 172.21.15.77:49994) 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ecd519700 1 -- 172.21.15.77:0/2214497337 learned_addr learned my addr 172.21.15.77:0/2214497337 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f6ece51b700 1 --2- 172.21.15.77:0/2214497337 >> v2:172.21.15.77:3300/0 conn(0x7f6ec80ad700 0x7f6ec80adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 -- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 msgr2=0x7f6ec8164f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 --2- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec8164f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 -- 172.21.15.77:0/2214497337 >> v2:172.21.15.77:3300/0 conn(0x7f6ec80ad700 msgr2=0x7f6ec80adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 --2- 172.21.15.77:0/2214497337 >> v2:172.21.15.77:3300/0 conn(0x7f6ec80ad700 0x7f6ec80adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 -- 172.21.15.77:0/2214497337 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80ce1b0 con 0x7f6ec80cd730 2023-07-04T18:51:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f6ecd519700 1 --2- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec806bca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806f140 tx=0x7f6eb8009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c828754e8fbec0e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f03dc9d3700 1 Processor -- start 2023-07-04T18:51:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.688+0000 7f03dc9d3700 1 -- start start 2023-07-04T18:51:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03dc9d3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03d80c5d30 0x7f03d80b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03dc9d3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d80b46d0 con 0x7f03d8160c70 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03dc9d3700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d80d0a90 con 0x7f03d80d33c0 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03dc9d3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d8161040 con 0x7f03d80c5d30 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03d6d9d700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f03d80d33c0 0x7f03d80afa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59232/0 (socket says 172.21.15.77:59232) 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03d6d9d700 1 -- 172.21.15.77:0/3478369510 learned_addr learned my addr 172.21.15.77:0/3478369510 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.690+0000 7f03d659c700 1 --1- 172.21.15.77:0/3478369510 >> v1:172.21.15.77:6789/0 conn(0x7f03d8160c70 0x7f03d80d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51316/0 (socket says 172.21.15.77:51316) 2023-07-04T18:51:28.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.691+0000 7f03d5d9b700 1 --2- 172.21.15.77:0/3478369510 >> v2:172.21.15.77:3300/0 conn(0x7f03d80c5d30 0x7f03d80b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0415c2e0 con 0x7f0c040b0b50 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.689+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c040c7bb0 con 0x7f0c040b0b50 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0bfc019070 con 0x7f0c040b0b50 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f0bfa7fc700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 0x7f0be806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0bfc08a580 con 0x7f0c040b0b50 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f0c09636700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 0x7f0be806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec0052f0 con 0x7f0c040b0b50 2023-07-04T18:51:28.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7fe65259c700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 0x7fe63406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.692+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90013680 con 0x7f2a941fe830 2023-07-04T18:51:28.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a940b36e0 con 0x7f2a941fe830 2023-07-04T18:51:28.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe64405c920 con 0x7fe6540ce320 2023-07-04T18:51:28.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a941feca0 con 0x7f2a941fe830 2023-07-04T18:51:28.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7fe65259c700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 0x7fe63406d210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe648001eb0 tx=0x7fe648001580 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.694+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a900137e0 con 0x7f2a941fe830 2023-07-04T18:51:28.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.694+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90013b90 con 0x7f2a941fe830 2023-07-04T18:51:28.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.694+0000 7f16a5471700 1 Processor -- start 2023-07-04T18:51:28.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.694+0000 7f16a5471700 1 -- start start 2023-07-04T18:51:28.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.694+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a940b3c00 con 0x7f2a941fe830 2023-07-04T18:51:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f03d5d9b700 1 -- 172.21.15.77:0/3478369510 >> v1:172.21.15.82:6789/0 conn(0x7f03d80d33c0 legacy=0x7f03d80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f03d5d9b700 1 -- 172.21.15.77:0/3478369510 >> v1:172.21.15.77:6789/0 conn(0x7f03d8160c70 legacy=0x7f03d80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f03d559a700 1 -- 172.21.15.77:0/3478369510 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372641444 0 0) 0x7f03d80b46d0 con 0x7f03d8160c70 2023-07-04T18:51:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f03d5d9b700 1 -- 172.21.15.77:0/3478369510 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d80d0d70 con 0x7f03d80c5d30 2023-07-04T18:51:28.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.693+0000 7f03d5d9b700 1 --2- 172.21.15.77:0/3478369510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d80b40f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009840 tx=0x7f03cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4593542280ce0d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f0c09636700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 0x7f0be806d260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cea40 tx=0x7f0c00008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f6eccd18700 1 -- 172.21.15.77:0/2214497337 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb800b870 con 0x7f6ec80cd730 2023-07-04T18:51:28.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f6eccd18700 1 -- 172.21.15.77:0/2214497337 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb800b9d0 con 0x7f6ec80cd730 2023-07-04T18:51:28.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f6eccd18700 1 -- 172.21.15.77:0/2214497337 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb800bd60 con 0x7f6ec80cd730 2023-07-04T18:51:28.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f65ac17a8a0 con 0x7f65ac06b120 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004fc0 0x7f16a01fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16a00b3b60 0x7f16a00c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00c43c0 con 0x7f16a00c23e0 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01ffcd0 con 0x7f16a00b3b60 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f16a5471700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01ff430 con 0x7f1688004fc0 2023-07-04T18:51:28.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f169f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f169f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44198/0 (socket says 172.21.15.77:44198) 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f169f7fe700 1 -- 172.21.15.77:0/2236487020 learned_addr learned my addr 172.21.15.77:0/2236487020 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.695+0000 7f169e7fc700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16a00b3b60 0x7f16a00c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 -- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004fc0 msgr2=0x7f16a01fed30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004fc0 0x7f16a01fed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 -- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16a00b3b60 msgr2=0x7f16a00c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16a00b3b60 0x7f16a00c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 -- 172.21.15.77:0/2236487020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a01ff710 con 0x7f16a00c23e0 2023-07-04T18:51:28.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f169f7fe700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f16a00cf920 tx=0x7f169800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 -- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 msgr2=0x7f6ec806bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec806bca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806f140 tx=0x7f6eb8009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 -- 172.21.15.77:0/2214497337 shutdown_connections 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/2214497337 >> v2:172.21.15.77:3300/0 conn(0x7f6ec80ad700 0x7f6ec80adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec8164f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/2214497337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec806bca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 -- 172.21.15.77:0/2214497337 >> 172.21.15.77:0/2214497337 conn(0x7f6ec806aad0 msgr2=0x7f6ec80b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 -- 172.21.15.77:0/2214497337 shutdown_connections 2023-07-04T18:51:28.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.696+0000 7f6ecff7e700 1 -- 172.21.15.77:0/2214497337 wait complete. 2023-07-04T18:51:28.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.697+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0bfc020080 con 0x7f0c040b0b50 2023-07-04T18:51:28.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.698+0000 7f6ecff7e700 1 Processor -- start 2023-07-04T18:51:28.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.698+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe644054ca0 con 0x7fe6540ce320 2023-07-04T18:51:28.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.698+0000 7f6ecff7e700 1 -- start start 2023-07-04T18:51:28.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.698+0000 7f03d559a700 1 -- 172.21.15.77:0/3478369510 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc011830 con 0x7f03d80c5d30 2023-07-04T18:51:28.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec80ad700 0x7f6ec80cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec80c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ccb00 con 0x7f6ec80ad700 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80cb560 con 0x7f6ec80cd730 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecff7e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80cb800 con 0x7f6ec8160ab0 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecd519700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecd519700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50008/0 (socket says 172.21.15.77:50008) 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.699+0000 7f6ecd519700 1 -- 172.21.15.77:0/671555573 learned_addr learned my addr 172.21.15.77:0/671555573 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec80c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698004d80 con 0x7f16a00c23e0 2023-07-04T18:51:28.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1698007110 con 0x7f16a00c23e0 2023-07-04T18:51:28.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16980074e0 con 0x7f16a00c23e0 2023-07-04T18:51:28.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a01ff9f0 con 0x7f16a00c23e0 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 -- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 msgr2=0x7f6ec80cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 -- 172.21.15.77:0/671555573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec80ad700 msgr2=0x7f6ec80cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec80ad700 0x7f6ec80cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ece51b700 1 -- 172.21.15.77:0/671555573 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8009550 con 0x7f6ec8160ab0 2023-07-04T18:51:28.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f6ecd519700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2a90013cf0 con 0x7f2a941fe830 2023-07-04T18:51:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.701+0000 7f2a8a7fc700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 0x7f2a7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.700+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a00c2880 con 0x7f16a00c23e0 2023-07-04T18:51:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.701+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2a90089860 con 0x7f2a941fe830 2023-07-04T18:51:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.701+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a00b1ab0 con 0x7f16a00c23e0 2023-07-04T18:51:28.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ece51b700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec80c64c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d5a0 tx=0x7f6ec400d8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7fe72b503700 1 Processor -- start 2023-07-04T18:51:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7fe72b503700 1 -- start start 2023-07-04T18:51:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40125f0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec80cbae0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec4012750 con 0x7f6ec8160ab0 2023-07-04T18:51:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4012ae0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.702+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec80cc0a0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7f2a99c63700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 0x7f2a7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.703+0000 7fe72b503700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.703+0000 7fe72b503700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fe724070470 0x7fe7240708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.703+0000 7fe72b503700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72406bd30 con 0x7fe7240ca6e0 2023-07-04T18:51:28.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.703+0000 7fe72b503700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe724158f00 con 0x7fe724070470 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.703+0000 7fe72b503700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72406b120 con 0x7fe724158b30 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34058/0 (socket says 172.21.15.77:34058) 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 -- 172.21.15.77:0/957438287 learned_addr learned my addr 172.21.15.77:0/957438287 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe728a9e700 1 --2- 172.21.15.77:0/957438287 >> v2:172.21.15.82:3300/0 conn(0x7fe724070470 0x7fe7240708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 -- 172.21.15.77:0/957438287 >> v2:172.21.15.82:3300/0 conn(0x7fe724070470 msgr2=0x7fe7240708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:28.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 --2- 172.21.15.77:0/957438287 >> v2:172.21.15.82:3300/0 conn(0x7fe724070470 0x7fe7240708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 -- 172.21.15.77:0/957438287 >> v1:172.21.15.77:6789/0 conn(0x7fe7240ca6e0 legacy=0x7fe7240cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 -- 172.21.15.77:0/957438287 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72406b400 con 0x7fe724158b30 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe72929f700 1 --2- 172.21.15.77:0/957438287 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7200098f0 tx=0x7fe720009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8243374e0cd512 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.704+0000 7fe71bfff700 1 -- 172.21.15.77:0/957438287 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72000f940 con 0x7fe724158b30 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7fe71bfff700 1 -- 172.21.15.77:0/957438287 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72000faa0 con 0x7fe724158b30 2023-07-04T18:51:28.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7fe71bfff700 1 -- 172.21.15.77:0/957438287 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72000fe30 con 0x7fe724158b30 2023-07-04T18:51:28.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2a900548f0 con 0x7f2a941fe830 2023-07-04T18:51:28.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6ec4012c40 con 0x7f6ec8160ab0 2023-07-04T18:51:28.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7f6ebe7fc700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 0x7f6eac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7fe72b503700 1 -- 172.21.15.77:0/957438287 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 msgr2=0x7fe7240cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7fe72b503700 1 --2- 172.21.15.77:0/957438287 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7200098f0 tx=0x7fe720009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:28.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.705+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6ec40894c0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7f6ecdd1a700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 0x7f6eac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7f03d559a700 1 -- 172.21.15.77:0/3478369510 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc011990 con 0x7f03d80c5d30 2023-07-04T18:51:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7f03d559a700 1 -- 172.21.15.77:0/3478369510 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc011d20 con 0x7f03d80c5d30 2023-07-04T18:51:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7fe72b503700 1 -- 172.21.15.77:0/957438287 shutdown_connections 2023-07-04T18:51:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7fe72b503700 1 --2- 172.21.15.77:0/957438287 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe724158b30 0x7fe7240cc480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7fe72b503700 1 --2- 172.21.15.77:0/957438287 >> v2:172.21.15.82:3300/0 conn(0x7fe724070470 0x7fe7240708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.706+0000 7fe72b503700 1 -- 172.21.15.77:0/957438287 >> 172.21.15.77:0/957438287 conn(0x7fe72406aad0 msgr2=0x7fe7240b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f03dc9d3700 1 -- 172.21.15.77:0/3478369510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 msgr2=0x7f03d80b40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/3478369510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d80b40f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009840 tx=0x7f03cc009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f03dc9d3700 1 -- 172.21.15.77:0/3478369510 shutdown_connections 2023-07-04T18:51:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/3478369510 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d80b40f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f03dc9d3700 1 -- 172.21.15.77:0/3478369510 >> 172.21.15.77:0/3478369510 conn(0x7f03d8071e90 msgr2=0x7f03d809c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.707+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec80cc2c0 con 0x7f6ec8160ab0 2023-07-04T18:51:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.708+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1698007640 con 0x7f16a00c23e0 2023-07-04T18:51:28.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.708+0000 7fe72b503700 1 -- 172.21.15.77:0/957438287 shutdown_connections 2023-07-04T18:51:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.708+0000 7fe72b503700 1 -- 172.21.15.77:0/957438287 wait complete. 2023-07-04T18:51:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.710+0000 7fe72b503700 1 Processor -- start 2023-07-04T18:51:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.708+0000 7f16877fe700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 0x7f168c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.708+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f16980891f0 con 0x7f16a00c23e0 2023-07-04T18:51:28.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.710+0000 7fe72b503700 1 -- start start 2023-07-04T18:51:28.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe724158b30 0x7fe7241506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7240cd870 0x7fe7240cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240b3410 con 0x7fe7240cd870 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7240ce210 con 0x7fe724158b30 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72b503700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe724159580 con 0x7fe7240ca6e0 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.711+0000 7fe72929f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34066/0 (socket says 172.21.15.77:34066) 2023-07-04T18:51:28.775 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 -- 172.21.15.77:0/3858735544 learned_addr learned my addr 172.21.15.77:0/3858735544 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe729aa0700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7240cd870 0x7fe7240cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 -- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe724158b30 msgr2=0x7fe7241506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe724158b30 0x7fe7241506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 -- 172.21.15.77:0/3858735544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7240cd870 msgr2=0x7fe7240cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7240cd870 0x7fe7240cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 -- 172.21.15.77:0/3858735544 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720009550 con 0x7fe7240ca6e0 2023-07-04T18:51:28.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72929f700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe72000b0e0 tx=0x7fe720004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7200038a0 con 0x7fe7240ca6e0 2023-07-04T18:51:28.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.713+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6ec4054560 con 0x7f6ec8160ab0 2023-07-04T18:51:28.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe720003a00 con 0x7fe7240ca6e0 2023-07-04T18:51:28.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe72b503700 1 -- 172.21.15.77:0/3858735544 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7240b27b0 con 0x7fe7240ca6e0 2023-07-04T18:51:28.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.712+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720003db0 con 0x7fe7240ca6e0 2023-07-04T18:51:28.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.713+0000 7f2a99c63700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 0x7f2a7806d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a84000c00 tx=0x7f2a84008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.778 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.714+0000 7f6ecdd1a700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 0x7f6eac06d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000abb0 tx=0x7f6ec0009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.714+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1698054410 con 0x7f16a00c23e0 2023-07-04T18:51:28.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.714+0000 7f03dc9d3700 1 -- 172.21.15.77:0/3478369510 shutdown_connections 2023-07-04T18:51:28.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.714+0000 7f03dc9d3700 1 -- 172.21.15.77:0/3478369510 wait complete. 2023-07-04T18:51:28.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.713+0000 7fe72b503700 1 -- 172.21.15.77:0/3858735544 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7240b2d50 con 0x7fe7240ca6e0 2023-07-04T18:51:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.716+0000 7f03dc9d3700 1 Processor -- start 2023-07-04T18:51:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.717+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7240b3550 con 0x7fe7240ca6e0 2023-07-04T18:51:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.717+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe720020440 con 0x7fe7240ca6e0 2023-07-04T18:51:28.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.717+0000 7fe719ffb700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 0x7fe70806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.716+0000 7f03dc9d3700 1 -- start start 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.717+0000 7f03dc9d3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d815db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03dc9d3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 0x7f03d80c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03dc9d3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 0x7f03d814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03dc9d3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815cb50 con 0x7f03d80c5d30 2023-07-04T18:51:28.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03dc9d3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815d6d0 con 0x7f03d80d33c0 2023-07-04T18:51:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03dc9d3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815cda0 con 0x7f03d8160c70 2023-07-04T18:51:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe720013070 con 0x7fe7240ca6e0 2023-07-04T18:51:28.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7fe728a9e700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 0x7fe70806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d5d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 0x7f03d80c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d5d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 0x7f03d80c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50034/0 (socket says 172.21.15.77:50034) 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d5d9b700 1 -- 172.21.15.77:0/4258137995 learned_addr learned my addr 172.21.15.77:0/4258137995 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 0x7f03d814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d659c700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d815db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 -- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 msgr2=0x7f03d80c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 0x7f03d80c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 -- 172.21.15.77:0/4258137995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 msgr2=0x7f03d815db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d815db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.718+0000 7f03d6d9d700 1 -- 172.21.15.77:0/4258137995 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0094a0 con 0x7f03d8160c70 2023-07-04T18:51:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.719+0000 7fe728a9e700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 0x7fe70806d190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b1ec0 tx=0x7fe714008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.784 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.719+0000 7f03d6d9d700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 0x7f03d814cc10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03d000ca70 tx=0x7f03d000cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.721+0000 7f169effd700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 0x7f168c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.721+0000 7f169effd700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 0x7f168c06d1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16a008a320 tx=0x7f1690006c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.722+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe7240b3550 con 0x7fe7240ca6e0 2023-07-04T18:51:28.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.727+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000a9a0 con 0x7f03d8160c70 2023-07-04T18:51:28.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.727+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d000ab00 con 0x7f03d8160c70 2023-07-04T18:51:28.786 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.727+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000ae90 con 0x7f03d8160c70 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.727+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d815d080 con 0x7f03d8160c70 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.727+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d80c5480 con 0x7f03d8160c70 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.730+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03d00113a0 con 0x7f03d8160c70 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.730+0000 7f03c6ffd700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 0x7f03b806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.731+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f03d0088f20 con 0x7f03d8160c70 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.731+0000 7f03d659c700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 0x7f03b806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:28.787 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.732+0000 7f03d659c700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 0x7f03b806d260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03c8000f60 tx=0x7f03c8007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:28.788 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.738+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d804e210 con 0x7f03d8160c70 2023-07-04T18:51:28.788 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:28.744+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f03d0054040 con 0x7f03d8160c70 2023-07-04T18:51:29.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.085+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe65404e210 con 0x7fe6540ce320 2023-07-04T18:51:29.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.089+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f65ac17a8a0 con 0x7f65ac06b120 2023-07-04T18:51:29.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.090+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6ec804e210 con 0x7f6ec8160ab0 2023-07-04T18:51:29.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.091+0000 7f65a2ffd700 1 -- 172.21.15.77:0/3499153335 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65ac17a8a0 con 0x7f65ac06b120 2023-07-04T18:51:29.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.091+0000 7f6ebe7fc700 1 -- 172.21.15.77:0/671555573 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ec4057b80 con 0x7f6ec8160ab0 2023-07-04T18:51:29.092 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:29.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.091+0000 7fe642ffd700 1 -- 172.21.15.77:0/789334337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe6440582c0 con 0x7fe6540ce320 2023-07-04T18:51:29.092 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:29.094 INFO:teuthology.orchestra.run.smithi077.stdout:55834574853 2023-07-04T18:51:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.095+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 msgr2=0x7f6eac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.095+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 0x7f6eac06d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000abb0 tx=0x7f6ec0009250 comp rx=0 tx=0).stop 2023-07-04T18:51:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.095+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 msgr2=0x7f6ec80c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.095+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec80c64c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400d5a0 tx=0x7f6ec400d8b0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.096+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0bec0055a0 con 0x7f0c040b0b50 2023-07-04T18:51:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 msgr2=0x7fe63406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 0x7fe63406d210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe648001eb0 tx=0x7fe648001580 comp rx=0 tx=0).stop 2023-07-04T18:51:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 msgr2=0x7fe6540c23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe644009da0 tx=0x7fe644004790 comp rx=0 tx=0).stop 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 shutdown_connections 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe63406ad60 0x7fe63406d210 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe6540c4760 0x7fe6540c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe63c0043e0 0x7fe6540c4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 --2- 172.21.15.77:0/789334337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6540ce320 0x7fe6540c23e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 >> 172.21.15.77:0/789334337 conn(0x7fe65406aad0 msgr2=0x7fe6540af470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 shutdown_connections 2023-07-04T18:51:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 msgr2=0x7f659006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.101 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 0x7f659006d1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f65ac0ad7f0 tx=0x7f65a8008040 comp rx=0 tx=0).stop 2023-07-04T18:51:29.101 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 msgr2=0x7f65ac20f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.101 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f65a400cc50 tx=0x7f65a400cf60 comp rx=0 tx=0).stop 2023-07-04T18:51:29.101 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7fe658922700 1 -- 172.21.15.77:0/789334337 wait complete. 2023-07-04T18:51:29.102 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 shutdown_connections 2023-07-04T18:51:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6eac06ace0 0x7f6eac06d190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ec80ad700 0x7f6ec80cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6ec8160ab0 0x7f6ec80c64c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 --2- 172.21.15.77:0/671555573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6ec80cd730 0x7f6ec80cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 >> 172.21.15.77:0/671555573 conn(0x7f6ec806aad0 msgr2=0x7f6ec80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 shutdown_connections 2023-07-04T18:51:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.097+0000 7f6ecff7e700 1 -- 172.21.15.77:0/671555573 wait complete. 2023-07-04T18:51:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f0bfa7fc700 1 -- 172.21.15.77:0/57543037 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0bfc017070 con 0x7f0c040b0b50 2023-07-04T18:51:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 shutdown_connections 2023-07-04T18:51:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f659006ad30 0x7f659006d1e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f65ac16cdd0 0x7f65ac0c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65ac06b120 0x7f65ac20f1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 --2- 172.21.15.77:0/3499153335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65ac0cdc70 0x7f65ac0b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 >> 172.21.15.77:0/3499153335 conn(0x7f65ac06aad0 msgr2=0x7f65ac0aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.098+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 shutdown_connections 2023-07-04T18:51:29.106 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:29.107 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f65b4348700 1 -- 172.21.15.77:0/3499153335 wait complete. 2023-07-04T18:51:29.107 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 msgr2=0x7f0be806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.107 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 0x7f0be806d260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cea40 tx=0x7f0c00008040 comp rx=0 tx=0).stop 2023-07-04T18:51:29.107 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 msgr2=0x7f0c040cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b370 tx=0x7f0bfc004ad0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 shutdown_connections 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f0be806adb0 0x7f0be806d260 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c040cea40 tx=0x7f0c00008040 comp rx=0 tx=0).stop 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c0414dcc0 0x7f0c04145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0c040b0b50 0x7f0c040cb4e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 --2- 172.21.15.77:0/57543037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0c040cc8d0 0x7f0c0414d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.102+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 >> 172.21.15.77:0/57543037 conn(0x7f0c0406aad0 msgr2=0x7f0c040af130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.108+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2a940b3c00 con 0x7f2a941fe830 2023-07-04T18:51:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.109+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 shutdown_connections 2023-07-04T18:51:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.110+0000 7f2a8a7fc700 1 -- 172.21.15.77:0/1671931857 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2a940b3c00 con 0x7f2a941fe830 2023-07-04T18:51:29.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.110+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe7240b3550 con 0x7fe7240ca6e0 2023-07-04T18:51:29.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.111+0000 7f0c0c09b700 1 -- 172.21.15.77:0/57543037 wait complete. 2023-07-04T18:51:29.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.111+0000 7fe719ffb700 1 -- 172.21.15.77:0/3858735544 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7240b3550 con 0x7fe7240ca6e0 2023-07-04T18:51:29.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.114+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f16a00b1ab0 con 0x7f16a00c23e0 2023-07-04T18:51:29.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.114+0000 7f16877fe700 1 -- 172.21.15.77:0/2236487020 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16a00b1ab0 con 0x7f16a00c23e0 2023-07-04T18:51:29.115 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:29.115 INFO:teuthology.orchestra.run.smithi077.stdout:55834574853 2023-07-04T18:51:29.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.117+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 msgr2=0x7f168c06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.117+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 0x7f168c06d1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16a008a320 tx=0x7f1690006c60 comp rx=0 tx=0).stop 2023-07-04T18:51:29.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.117+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 msgr2=0x7f16a00c7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.117+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f16a00cf920 tx=0x7f169800ccd0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.118 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.118+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 msgr2=0x7f2a7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.118+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 0x7f2a7806d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a84000c00 tx=0x7f2a84008030 comp rx=0 tx=0).stop 2023-07-04T18:51:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.118+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 msgr2=0x7f2a94153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.118+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 0x7f2a94153770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a9406dfd0 tx=0x7f2a9000eae0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.120 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-07-04T18:51:29.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-07-04T18:51:29.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 shutdown_connections 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2a7806ace0 0x7f2a7806d190 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2a94153cb0 0x7f2a940b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a80003a00 0x7f2a940b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 --2- 172.21.15.77:0/1671931857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2a941fe830 0x7f2a94153770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 >> 172.21.15.77:0/1671931857 conn(0x7f2a9406aad0 msgr2=0x7f2a940cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 shutdown_connections 2023-07-04T18:51:29.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.119+0000 7f2a9bec7700 1 -- 172.21.15.77:0/1671931857 wait complete. 2023-07-04T18:51:29.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.121+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 msgr2=0x7fe70806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.121+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 0x7fe70806d190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe7240b1ec0 tx=0x7fe714008040 comp rx=0 tx=0).stop 2023-07-04T18:51:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.121+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 msgr2=0x7fe724159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.121+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe72000b0e0 tx=0x7fe720004750 comp rx=0 tx=0).stop 2023-07-04T18:51:29.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 shutdown_connections 2023-07-04T18:51:29.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe70806ace0 0x7fe70806d190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7240cd870 0x7fe7240cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe7240ca6e0 0x7fe724159040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 --2- 172.21.15.77:0/3858735544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe724158b30 0x7fe7241506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 >> 172.21.15.77:0/3858735544 conn(0x7fe72406aad0 msgr2=0x7fe72406fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 shutdown_connections 2023-07-04T18:51:29.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.123+0000 7fe70f7fe700 1 -- 172.21.15.77:0/3858735544 wait complete. 2023-07-04T18:51:29.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 shutdown_connections 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f168c06ad30 0x7f168c06d1e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16a00c23e0 0x7f16a00c7b50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1688004fc0 0x7f16a01fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 --2- 172.21.15.77:0/2236487020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16a00b3b60 0x7f16a00c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 >> 172.21.15.77:0/2236487020 conn(0x7f16a006aad0 msgr2=0x7f16a00af440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 shutdown_connections 2023-07-04T18:51:29.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.125+0000 7f16a5471700 1 -- 172.21.15.77:0/2236487020 wait complete. 2023-07-04T18:51:29.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.132+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f03d815d810 con 0x7f03d8160c70 2023-07-04T18:51:29.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.133+0000 7f03c6ffd700 1 -- 172.21.15.77:0/4258137995 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03d0017070 con 0x7f03d8160c70 2023-07-04T18:51:29.136 INFO:teuthology.orchestra.run.smithi077.stdout:55834574853 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 msgr2=0x7f03b806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 0x7f03b806d260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f03c8000f60 tx=0x7f03c8007e70 comp rx=0 tx=0).stop 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 msgr2=0x7f03d814cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 0x7f03d814cc10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f03d000ca70 tx=0x7f03d000cd80 comp rx=0 tx=0).stop 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 shutdown_connections 2023-07-04T18:51:29.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b806adb0 0x7f03b806d260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d80c5d30 0x7f03d815db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d8160c70 0x7f03d814cc10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 --2- 172.21.15.77:0/4258137995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d80d33c0 0x7f03d80c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.135+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 >> 172.21.15.77:0/4258137995 conn(0x7f03d8071e90 msgr2=0x7f03d80b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.136+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 shutdown_connections 2023-07-04T18:51:29.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:29.136+0000 7f03dc9d3700 1 -- 172.21.15.77:0/4258137995 wait complete. 2023-07-04T18:51:29.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2023-07-04T18:51:29.142 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-07-04T18:51:29.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2023-07-04T18:51:29.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-07-04T18:51:29.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2023-07-04T18:51:30.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-04T18:51:30.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-07-04T18:51:30.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-07-04T18:51:30.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-07-04T18:51:30.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-07-04T18:51:30.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-04T18:51:30.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-04T18:51:30.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-04T18:51:30.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.328+0000 7fc3fd634700 1 Processor -- start 2023-07-04T18:51:30.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.328+0000 7fc3fd634700 1 -- start start 2023-07-04T18:51:30.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.329+0000 7fc3fd634700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f815be90 0x7fc3f80cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.329+0000 7fc3fd634700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f815c260 con 0x7fc3f80ce250 2023-07-04T18:51:30.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.329+0000 7fc3fd634700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80ca690 con 0x7fc3f80ad700 2023-07-04T18:51:30.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.329+0000 7fc3fd634700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80ca970 con 0x7fc3f815be90 2023-07-04T18:51:30.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.330+0000 7fc3f77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc3f80ce250 0x7fc3f815c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51372/0 (socket says 172.21.15.77:51372) 2023-07-04T18:51:30.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.330+0000 7fc3f77fe700 1 -- 172.21.15.77:0/688657972 learned_addr learned my addr 172.21.15.77:0/688657972 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:30.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.330+0000 7fc3f6ffd700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f815be90 0x7fc3f80cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891731289 0 0) 0x7fc3f815c260 con 0x7fc3f80ce250 2023-07-04T18:51:30.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e00035b0 con 0x7fc3f80ce250 2023-07-04T18:51:30.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1692441644 0 0) 0x7fc3e00035b0 con 0x7fc3f80ce250 2023-07-04T18:51:30.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f815be90 msgr2=0x7fc3f80cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f815be90 0x7fc3f80cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 >> v1:172.21.15.82:6789/0 conn(0x7fc3f80ad700 legacy=0x7fc3f80cdb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80cac50 con 0x7fc3f80ce250 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fc3f00019f0 con 0x7fc3f80ce250 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 >> v1:172.21.15.77:6789/0 conn(0x7fc3f80ce250 legacy=0x7fc3f815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e0003a30 0x7fc3e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3e00043f0 0x7fc3e00048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3e00050b0 0x7fc3e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80ca970 con 0x7fc3e0003a30 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e000bf80 con 0x7fc3e00043f0 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e000c140 con 0x7fc3e00050b0 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.331+0000 7fc3f77fe700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3e00050b0 0x7fc3e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f67fc700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3e00043f0 0x7fc3e00048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f6ffd700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e0003a30 0x7fc3e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 -- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3e00043f0 msgr2=0x7fc3e00048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3e00043f0 0x7fc3e00048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 -- 172.21.15.77:0/688657972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e0003a30 msgr2=0x7fc3e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e0003a30 0x7fc3e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 -- 172.21.15.77:0/688657972 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80cac50 con 0x7fc3e00050b0 2023-07-04T18:51:30.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f77fe700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3e00050b0 0x7fc3e000ba10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0008e10 tx=0x7fc3e0005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481b24cabbad9298 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.332+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0008310 con 0x7fc3e00050b0 2023-07-04T18:51:30.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.333+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0008e40 con 0x7fc3e00050b0 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.333+0000 7fc3f5ffb700 1 -- 172.21.15.77:0/688657972 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00149b0 con 0x7fc3e00050b0 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.333+0000 7fc3fd634700 1 -- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3e00050b0 msgr2=0x7fc3e000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.333+0000 7fc3fd634700 1 --2- 172.21.15.77:0/688657972 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3e00050b0 0x7fc3e000ba10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0008e10 tx=0x7fc3e0005880 comp rx=0 tx=0).stop 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.334+0000 7fc3f77fe700 1 -- 172.21.15.77:0/688657972 reap_dead start 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.334+0000 7fc3fd634700 1 -- 172.21.15.77:0/688657972 shutdown_connections 2023-07-04T18:51:30.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.334+0000 7fc3fd634700 1 -- 172.21.15.77:0/688657972 >> 172.21.15.77:0/688657972 conn(0x7fc3f806aad0 msgr2=0x7fc3f80b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.334+0000 7f553f2f4700 1 Processor -- start 2023-07-04T18:51:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.334+0000 7f553f2f4700 1 -- start start 2023-07-04T18:51:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.335+0000 7f553f2f4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553f2f4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553815c260 con 0x7f55380ce250 2023-07-04T18:51:30.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553f2f4700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55380ca690 con 0x7f55380ad700 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553f2f4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55380ca970 con 0x7f553815be90 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d090700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d891700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f55380ce250 0x7f553815c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51386/0 (socket says 172.21.15.77:51386) 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d891700 1 -- 172.21.15.77:0/2817525586 learned_addr learned my addr 172.21.15.77:0/2817525586 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d090700 1 --2- 172.21.15.77:0/2817525586 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34096/0 (socket says 172.21.15.77:34096) 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d090700 1 -- 172.21.15.77:0/2817525586 >> v1:172.21.15.82:6789/0 conn(0x7f55380ad700 legacy=0x7f55380cdb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:30.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d090700 1 -- 172.21.15.77:0/2817525586 >> v1:172.21.15.77:6789/0 conn(0x7f55380ce250 legacy=0x7f553815c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.336+0000 7f553d090700 1 -- 172.21.15.77:0/2817525586 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55380cac50 con 0x7f553815be90 2023-07-04T18:51:30.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7f553d090700 1 --2- 172.21.15.77:0/2817525586 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5534010be0 tx=0x7f5534010ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285a51f9e20f4cf5 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7fc3fd634700 1 -- 172.21.15.77:0/688657972 shutdown_connections 2023-07-04T18:51:30.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7fc3fd634700 1 -- 172.21.15.77:0/688657972 wait complete. 2023-07-04T18:51:30.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7f552ffff700 1 -- 172.21.15.77:0/2817525586 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553400be20 con 0x7f553815be90 2023-07-04T18:51:30.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7f552ffff700 1 -- 172.21.15.77:0/2817525586 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5534004030 con 0x7f553815be90 2023-07-04T18:51:30.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.337+0000 7f552ffff700 1 -- 172.21.15.77:0/2817525586 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340043a0 con 0x7f553815be90 2023-07-04T18:51:30.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.338+0000 7f553f2f4700 1 -- 172.21.15.77:0/2817525586 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 msgr2=0x7f55380cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.338+0000 7f553f2f4700 1 --2- 172.21.15.77:0/2817525586 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5534010be0 tx=0x7f5534010ef0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.339+0000 7f553f2f4700 1 -- 172.21.15.77:0/2817525586 shutdown_connections 2023-07-04T18:51:30.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.339+0000 7f553f2f4700 1 --2- 172.21.15.77:0/2817525586 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815be90 0x7f55380cc480 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5534010be0 tx=0x7f5534010ef0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.339+0000 7f553f2f4700 1 -- 172.21.15.77:0/2817525586 >> 172.21.15.77:0/2817525586 conn(0x7f553806aad0 msgr2=0x7f55380b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.339+0000 7fc3fd634700 1 Processor -- start 2023-07-04T18:51:30.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.340+0000 7fc3fd634700 1 -- start start 2023-07-04T18:51:30.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7f553f2f4700 1 -- 172.21.15.77:0/2817525586 shutdown_connections 2023-07-04T18:51:30.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3f81e3550 0x7fc3f814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f80c44d0 0x7fc3f80c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81e3ec0 con 0x7fc3e00050b0 2023-07-04T18:51:30.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81e39c0 con 0x7fc3f81e3550 2023-07-04T18:51:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3fd634700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81e1bc0 con 0x7fc3f80c44d0 2023-07-04T18:51:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44256/0 (socket says 172.21.15.77:44256) 2023-07-04T18:51:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7fc3f6ffd700 1 -- 172.21.15.77:0/2275544221 learned_addr learned my addr 172.21.15.77:0/2275544221 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 -- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f80c44d0 msgr2=0x7fc3f80c23e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f80c44d0 0x7fc3f80c23e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 -- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3f81e3550 msgr2=0x7fc3f814fef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3f81e3550 0x7fc3f814fef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 -- 172.21.15.77:0/2275544221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f81e3be0 con 0x7fc3e00050b0 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.342+0000 7fc3f6ffd700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec011590 tx=0x7fc3ec011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec015870 con 0x7fc3e00050b0 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ec0159d0 con 0x7fc3e00050b0 2023-07-04T18:51:30.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec015d80 con 0x7fc3e00050b0 2023-07-04T18:51:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f81e1570 con 0x7fc3e00050b0 2023-07-04T18:51:30.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f80c4940 con 0x7fc3e00050b0 2023-07-04T18:51:30.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.341+0000 7f553f2f4700 1 -- 172.21.15.77:0/2817525586 wait complete. 2023-07-04T18:51:30.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7f553f2f4700 1 Processor -- start 2023-07-04T18:51:30.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.343+0000 7f553f2f4700 1 -- start start 2023-07-04T18:51:30.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 0x7f55380702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815b240 0x7f55380707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380b2200 con 0x7f55380b32a0 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55380b24e0 con 0x7f55380ce250 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.344+0000 7f553f2f4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55380b27c0 con 0x7f553815b240 2023-07-04T18:51:30.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50064/0 (socket says 172.21.15.77:50064) 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 -- 172.21.15.77:0/4108227595 learned_addr learned my addr 172.21.15.77:0/4108227595 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553c88f700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 0x7f55380702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 -- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815b240 msgr2=0x7f55380707e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815b240 0x7f55380707e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 -- 172.21.15.77:0/4108227595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 msgr2=0x7f55380702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 0x7f55380702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7f553d090700 1 -- 172.21.15.77:0/4108227595 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534010810 con 0x7f55380ce250 2023-07-04T18:51:30.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc3ec0143e0 con 0x7fc3e00050b0 2023-07-04T18:51:30.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.345+0000 7fc3df7fe700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 0x7fc3e006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.346+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc3ec0893e0 con 0x7fc3e00050b0 2023-07-04T18:51:30.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.346+0000 7f553c88f700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 0x7f55380702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:30.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.350+0000 7fc3f67fc700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 0x7fc3e006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.350+0000 7f553d090700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f553806edb0 tx=0x7f5534004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.352+0000 7fc3f67fc700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 0x7fc3e006eb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8157b60 tx=0x7fc3e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.366 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.352+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d80052f0 con 0x7fc3e00050b0 2023-07-04T18:51:30.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.353+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534003ab0 con 0x7f55380ce250 2023-07-04T18:51:30.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.353+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5534003c30 con 0x7f55380ce250 2023-07-04T18:51:30.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.353+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534021460 con 0x7f55380ce250 2023-07-04T18:51:30.367 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.353+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55380b2aa0 con 0x7f55380ce250 2023-07-04T18:51:30.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.354+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55380c2620 con 0x7f55380ce250 2023-07-04T18:51:30.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.357+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f55340215c0 con 0x7f55380ce250 2023-07-04T18:51:30.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.358+0000 7f552dffb700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 0x7f551c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.358+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f553408b450 con 0x7f55380ce250 2023-07-04T18:51:30.368 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.361+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc3ec054490 con 0x7fc3e00050b0 2023-07-04T18:51:30.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.361+0000 7f553c88f700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 0x7f551c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.362+0000 7f553c88f700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 0x7f551c06d260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5538202c60 tx=0x7f5528008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.363+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55180052f0 con 0x7f55380ce250 2023-07-04T18:51:30.370 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.368+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f55340565a0 con 0x7f55380ce250 2023-07-04T18:51:30.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.370+0000 7f03d8bb7700 1 Processor -- start 2023-07-04T18:51:30.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.370+0000 7f03d8bb7700 1 -- start start 2023-07-04T18:51:30.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.371+0000 7f03d8bb7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03d415be90 0x7f03d40cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.371+0000 7f03d8bb7700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f03d40ad700 0x7f03d40adb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.371+0000 7f03d8bb7700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d415c260 con 0x7f03d40cdb40 2023-07-04T18:51:30.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.372+0000 7f03d8bb7700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40ca690 con 0x7f03d415be90 2023-07-04T18:51:30.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.372+0000 7f03d8bb7700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40ca970 con 0x7f03d40ad700 2023-07-04T18:51:30.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.372+0000 7f03d2d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f03d40cdb40 0x7f03d40cdfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51408/0 (socket says 172.21.15.77:51408) 2023-07-04T18:51:30.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.372+0000 7f03d2d9d700 1 -- 172.21.15.77:0/535360738 learned_addr learned my addr 172.21.15.77:0/535360738 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 --2- 172.21.15.77:0/535360738 >> v2:172.21.15.77:3300/0 conn(0x7f03d415be90 0x7f03d40cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d1d9b700 1 --2- 172.21.15.77:0/535360738 >> v2:172.21.15.82:3300/0 conn(0x7f03d40ad700 0x7f03d40adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 -- 172.21.15.77:0/535360738 >> v2:172.21.15.82:3300/0 conn(0x7f03d40ad700 msgr2=0x7f03d40adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 --2- 172.21.15.77:0/535360738 >> v2:172.21.15.82:3300/0 conn(0x7f03d40ad700 0x7f03d40adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 -- 172.21.15.77:0/535360738 >> v1:172.21.15.77:6789/0 conn(0x7f03d40cdb40 legacy=0x7f03d40cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 -- 172.21.15.77:0/535360738 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d40cac50 con 0x7f03d415be90 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.373+0000 7f03d259c700 1 --2- 172.21.15.77:0/535360738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d415be90 0x7f03d40cc480 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b11a0 tx=0x7f03c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc057bfb680afb60 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.376+0000 7f03d159a700 1 -- 172.21.15.77:0/535360738 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4004230 con 0x7f03d415be90 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.376+0000 7f03d159a700 1 -- 172.21.15.77:0/535360738 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4004390 con 0x7f03d415be90 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.376+0000 7f03d159a700 1 -- 172.21.15.77:0/535360738 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4010830 con 0x7f03d415be90 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.377+0000 7f03d8bb7700 1 -- 172.21.15.77:0/535360738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d415be90 msgr2=0x7f03d40cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.377+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/535360738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d415be90 0x7f03d40cc480 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b11a0 tx=0x7f03c4009b20 comp rx=0 tx=0).stop 2023-07-04T18:51:30.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.379+0000 7f9a90fdd700 1 Processor -- start 2023-07-04T18:51:30.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.379+0000 7f9a90fdd700 1 -- start start 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a90fdd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c15a600 0x7f9a8c0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a90fdd700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f9a8c0cb980 0x7f9a8c070340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a90fdd700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8c06b250 con 0x7f9a8c0ca590 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a90fdd700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a8c06bed0 con 0x7f9a8c0cb980 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a90fdd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c0caa00 con 0x7f9a8c15a600 2023-07-04T18:51:30.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f9a8c0cb980 0x7f9a8c070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f9a8c0cb980 0x7f9a8c070340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50080/0 (socket says 172.21.15.77:50080) 2023-07-04T18:51:30.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/255024594 learned_addr learned my addr 172.21.15.77:0/255024594 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c15a600 msgr2=0x7f9a8c0ca050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:30.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c15a600 0x7f9a8c0ca050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/255024594 >> v1:172.21.15.82:6789/0 conn(0x7f9a8c0ca590 legacy=0x7f9a8c0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.380+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/255024594 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c06b530 con 0x7f9a8c0cb980 2023-07-04T18:51:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0cb980 0x7f9a8c070340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f1076aa479647b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a8959a700 1 -- 172.21.15.77:0/255024594 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400e070 con 0x7f9a8c0cb980 2023-07-04T18:51:30.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a8959a700 1 -- 172.21.15.77:0/255024594 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a84003a20 con 0x7f9a8c0cb980 2023-07-04T18:51:30.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a8959a700 1 -- 172.21.15.77:0/255024594 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84003dd0 con 0x7f9a8c0cb980 2023-07-04T18:51:30.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a90fdd700 1 -- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0cb980 msgr2=0x7f9a8c070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.381+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0cb980 0x7f9a8c070340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009e50 tx=0x7f9a8400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:30.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.382+0000 7f9a90fdd700 1 -- 172.21.15.77:0/255024594 shutdown_connections 2023-07-04T18:51:30.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.382+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0cb980 0x7f9a8c070340 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.382+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/255024594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c15a600 0x7f9a8c0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.382+0000 7f9a90fdd700 1 -- 172.21.15.77:0/255024594 >> 172.21.15.77:0/255024594 conn(0x7f9a8c06aad0 msgr2=0x7f9a8c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7f03d8bb7700 1 -- 172.21.15.77:0/535360738 shutdown_connections 2023-07-04T18:51:30.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/535360738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d415be90 0x7f03d40cc480 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/535360738 >> v2:172.21.15.82:3300/0 conn(0x7f03d40ad700 0x7f03d40adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7f03d8bb7700 1 -- 172.21.15.77:0/535360738 >> 172.21.15.77:0/535360738 conn(0x7f03d406aad0 msgr2=0x7f03d40b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7f9a90fdd700 1 -- 172.21.15.77:0/255024594 shutdown_connections 2023-07-04T18:51:30.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.383+0000 7fc4e02c4700 1 Processor -- start 2023-07-04T18:51:30.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.384+0000 7fc4e02c4700 1 -- start start 2023-07-04T18:51:30.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4e02c4700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fc4d81592d0 0x7fc4d80ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4e02c4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc4d806c6f0 0x7fc4d80c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4e02c4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d80cd270 con 0x7fc4d80c8f00 2023-07-04T18:51:30.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4e02c4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d80c98e0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4e02c4700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d80cd550 con 0x7fc4d81592d0 2023-07-04T18:51:30.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4dd85f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc4d80c8f00 0x7fc4d815cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51418/0 (socket says 172.21.15.77:51418) 2023-07-04T18:51:30.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7fc4dd85f700 1 -- 172.21.15.77:0/4032593553 learned_addr learned my addr 172.21.15.77:0/4032593553 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:30.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7f03d8bb7700 1 -- 172.21.15.77:0/535360738 shutdown_connections 2023-07-04T18:51:30.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.385+0000 7f03d8bb7700 1 -- 172.21.15.77:0/535360738 wait complete. 2023-07-04T18:51:30.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.387+0000 7f03d8bb7700 1 Processor -- start 2023-07-04T18:51:30.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.387+0000 7f03d8bb7700 1 -- start start 2023-07-04T18:51:30.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d40cdb40 0x7f03d40c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d415be90 0x7f03d40c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40c3390 con 0x7f03d40ad700 2023-07-04T18:51:30.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40c2780 con 0x7f03d40cdb40 2023-07-04T18:51:30.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d8bb7700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40c2a60 con 0x7f03d415be90 2023-07-04T18:51:30.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44286/0 (socket says 172.21.15.77:44286) 2023-07-04T18:51:30.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 -- 172.21.15.77:0/1024835126 learned_addr learned my addr 172.21.15.77:0/1024835126 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d1d9b700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d40cdb40 0x7f03d40c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d2d9d700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d415be90 0x7f03d40c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 -- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d415be90 msgr2=0x7f03d40c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d415be90 0x7f03d40c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 -- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d40cdb40 msgr2=0x7f03d40c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d40cdb40 0x7f03d40c3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 -- 172.21.15.77:0/1024835126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40094a0 con 0x7f03d40ad700 2023-07-04T18:51:30.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.388+0000 7f03d259c700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03c401a040 tx=0x7f03c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de060700 1 --2- 172.21.15.77:0/4032593553 >> v2:172.21.15.82:3300/0 conn(0x7fc4d81592d0 0x7fc4d80ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 --2- 172.21.15.77:0/4032593553 >> v2:172.21.15.77:3300/0 conn(0x7fc4d806c6f0 0x7fc4d80c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7f9a90fdd700 1 -- 172.21.15.77:0/255024594 wait complete. 2023-07-04T18:51:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347539919 0 0) 0x7fc4d80cd270 con 0x7fc4d80c8f00 2023-07-04T18:51:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c40035b0 con 0x7fc4d80c8f00 2023-07-04T18:51:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230945691 0 0) 0x7fc4c40035b0 con 0x7fc4d80c8f00 2023-07-04T18:51:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 -- 172.21.15.77:0/4032593553 >> v2:172.21.15.82:3300/0 conn(0x7fc4d81592d0 msgr2=0x7fc4d80ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 --2- 172.21.15.77:0/4032593553 >> v2:172.21.15.82:3300/0 conn(0x7fc4d81592d0 0x7fc4d80ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 -- 172.21.15.77:0/4032593553 >> v1:172.21.15.77:6789/0 conn(0x7fc4d80c8f00 legacy=0x7fc4d815cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 -- 172.21.15.77:0/4032593553 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80cd830 con 0x7fc4d806c6f0 2023-07-04T18:51:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40041b0 con 0x7f03d40ad700 2023-07-04T18:51:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7fc4de861700 1 --2- 172.21.15.77:0/4032593553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d806c6f0 0x7fc4d80c93a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009d90 tx=0x7fc4d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49961bbd1597911 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.391+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000e070 con 0x7fc4d806c6f0 2023-07-04T18:51:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c40249d0 con 0x7f03d40ad700 2023-07-04T18:51:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.390+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4024d80 con 0x7f03d40ad700 2023-07-04T18:51:30.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.391+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0003a20 con 0x7fc4d806c6f0 2023-07-04T18:51:30.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.391+0000 7fc4dd05e700 1 -- 172.21.15.77:0/4032593553 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8001670 con 0x7fc4d806c6f0 2023-07-04T18:51:30.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7fc4e02c4700 1 -- 172.21.15.77:0/4032593553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d806c6f0 msgr2=0x7fc4d80c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/4032593553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d806c6f0 0x7fc4d80c93a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009d90 tx=0x7fc4d000b330 comp rx=0 tx=0).stop 2023-07-04T18:51:30.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7fc4e02c4700 1 -- 172.21.15.77:0/4032593553 shutdown_connections 2023-07-04T18:51:30.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/4032593553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d806c6f0 0x7fc4d80c93a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/4032593553 >> v2:172.21.15.82:3300/0 conn(0x7fc4d81592d0 0x7fc4d80ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7fc4e02c4700 1 -- 172.21.15.77:0/4032593553 >> 172.21.15.77:0/4032593553 conn(0x7fc4d806aad0 msgr2=0x7fc4d80b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7f9a90fdd700 1 Processor -- start 2023-07-04T18:51:30.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d40c2d40 con 0x7f03d40ad700 2023-07-04T18:51:30.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d414a140 con 0x7f03d40ad700 2023-07-04T18:51:30.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7f79ec5ae700 1 Processor -- start 2023-07-04T18:51:30.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.392+0000 7f79ec5ae700 1 -- start start 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79ec5ae700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f79e41585e0 0x7f79e40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79ec5ae700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79ec5ae700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e40cf220 con 0x7f79e40ced60 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79ec5ae700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e40cf500 con 0x7f79e41585e0 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79ec5ae700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40cf7e0 con 0x7f79e40cdbe0 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79e9b49700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79e9b49700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34142/0 (socket says 172.21.15.77:34142) 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f79e9b49700 1 -- 172.21.15.77:0/429463415 learned_addr learned my addr 172.21.15.77:0/429463415 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9b49700 1 -- 172.21.15.77:0/429463415 >> v2:172.21.15.77:3300/0 conn(0x7f79e41585e0 msgr2=0x7f79e40cd6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9b49700 1 --2- 172.21.15.77:0/429463415 >> v2:172.21.15.77:3300/0 conn(0x7f79e41585e0 0x7f79e40cd6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9b49700 1 -- 172.21.15.77:0/429463415 >> v1:172.21.15.77:6789/0 conn(0x7f79e40ced60 legacy=0x7f79e415c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9b49700 1 -- 172.21.15.77:0/429463415 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40cae10 con 0x7f79e40cdbe0 2023-07-04T18:51:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9b49700 1 --2- 172.21.15.77:0/429463415 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79e00092e0 tx=0x7f79e00095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30b32ff84558bc96 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f9a90fdd700 1 -- start start 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c0cb980 0x7f9a8c0c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 0x7f9a8c14beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 0x7f9a8c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c0c7330 con 0x7f9a8c0cc270 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c0ccc20 con 0x7f9a8c0c6ec0 2023-07-04T18:51:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a90fdd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c1507e0 con 0x7f9a8c0cb980 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a89d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 0x7f9a8c14beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a89d9b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 0x7f9a8c14beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50104/0 (socket says 172.21.15.77:50104) 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a89d9b700 1 -- 172.21.15.77:0/3734476055 learned_addr learned my addr 172.21.15.77:0/3734476055 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 0x7f9a8c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f9a8a59c700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c0cb980 0x7f9a8c0c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c0cb980 msgr2=0x7f9a8c0c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c0cb980 0x7f9a8c0c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 msgr2=0x7f9a8c14beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 0x7f9a8c14beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 -- 172.21.15.77:0/3734476055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84009b30 con 0x7f9a8c0cc270 2023-07-04T18:51:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f9a8ad9d700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 0x7f9a8c0cc6e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c06ef60 tx=0x7f9a84004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.393+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d40c3670 con 0x7f03d40ad700 2023-07-04T18:51:30.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03c40036a0 con 0x7f03d40ad700 2023-07-04T18:51:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9348700 1 -- 172.21.15.77:0/429463415 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000c9b0 con 0x7f79e40cdbe0 2023-07-04T18:51:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9348700 1 -- 172.21.15.77:0/429463415 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000cb10 con 0x7f79e40cdbe0 2023-07-04T18:51:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7f79e9348700 1 -- 172.21.15.77:0/429463415 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000cea0 con 0x7f79e40cdbe0 2023-07-04T18:51:30.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 -- 172.21.15.77:0/429463415 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 msgr2=0x7f79e40ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/429463415 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79e00092e0 tx=0x7f79e00095f0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 -- 172.21.15.77:0/429463415 shutdown_connections 2023-07-04T18:51:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/429463415 >> v2:172.21.15.77:3300/0 conn(0x7f79e41585e0 0x7f79e40cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/429463415 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40cdbe0 0x7f79e40ce180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f79ec5ae700 1 -- 172.21.15.77:0/429463415 >> 172.21.15.77:0/429463415 conn(0x7f79e4069e40 msgr2=0x7f79e406a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.394+0000 7fc4e02c4700 1 -- 172.21.15.77:0/4032593553 shutdown_connections 2023-07-04T18:51:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7fc4e02c4700 1 -- 172.21.15.77:0/4032593553 wait complete. 2023-07-04T18:51:30.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7fc4e02c4700 1 Processor -- start 2023-07-04T18:51:30.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7fc4e02c4700 1 -- start start 2023-07-04T18:51:30.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84004440 con 0x7f9a8c0cc270 2023-07-04T18:51:30.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a840092e0 con 0x7f9a8c0cc270 2023-07-04T18:51:30.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a840047e0 con 0x7f9a8c0cc270 2023-07-04T18:51:30.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f03c2ffd700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 0x7f03b406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f03c4092820 con 0x7f03d40ad700 2023-07-04T18:51:30.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.395+0000 7f03424e2700 1 Processor -- start 2023-07-04T18:51:30.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f03424e2700 1 -- start start 2023-07-04T18:51:30.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f03424e2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c157ac0 0x7f033c1643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f03424e2700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f033c0b1890 0x7f033c1648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f03424e2700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c070360 con 0x7f033c1576f0 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f03424e2700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033c0b3d70 con 0x7f033c0b1890 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f03424e2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c06f470 con 0x7f033c157ac0 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f033bfff700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f033c1576f0 0x7f033c0b3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59352/0 (socket says 172.21.15.77:59352) 2023-07-04T18:51:30.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f033bfff700 1 -- 172.21.15.77:0/2321370239 learned_addr learned my addr 172.21.15.77:0/2321370239 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:30.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.397+0000 7f0340a7f700 1 --2- 172.21.15.77:0/2321370239 >> v2:172.21.15.82:3300/0 conn(0x7f033c0b1890 0x7f033c1648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f5ad82ea700 1 Processor -- start 2023-07-04T18:51:30.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 0x7fc4d80c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d80c8f00 0x7fc4d82014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 0x7fc4d8201a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81e9c10 con 0x7fc4d80c8f00 2023-07-04T18:51:30.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81e9ef0 con 0x7fc4d81592d0 2023-07-04T18:51:30.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4e02c4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81ea1d0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de861700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 0x7fc4d8201a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 0x7fc4d80c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de861700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 0x7fc4d8201a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50114/0 (socket says 172.21.15.77:50114) 2023-07-04T18:51:30.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de861700 1 -- 172.21.15.77:0/2380621648 learned_addr learned my addr 172.21.15.77:0/2380621648 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c14fc40 con 0x7f9a8c0cc270 2023-07-04T18:51:30.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.396+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c14feb0 con 0x7f9a8c0cc270 2023-07-04T18:51:30.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f0340a7f700 1 -- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c157ac0 msgr2=0x7f033c1643a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:30.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f0340a7f700 1 --2- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c157ac0 0x7f033c1643a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f0340a7f700 1 -- 172.21.15.77:0/2321370239 >> v1:172.21.15.82:6789/0 conn(0x7f033c1576f0 legacy=0x7f033c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:30.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f0340a7f700 1 -- 172.21.15.77:0/2321370239 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c06f750 con 0x7f033c0b1890 2023-07-04T18:51:30.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a700052f0 con 0x7f9a8c0cc270 2023-07-04T18:51:30.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4dd85f700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d80c8f00 0x7fc4d82014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 -- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 msgr2=0x7fc4d8201a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 0x7fc4d8201a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 -- 172.21.15.77:0/2380621648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d80c8f00 msgr2=0x7fc4d82014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d80c8f00 0x7fc4d82014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7fc4de060700 1 -- 172.21.15.77:0/2380621648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0009a70 con 0x7fc4d806c6f0 2023-07-04T18:51:30.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f0340a7f700 1 --2- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c1648e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0330009840 tx=0x7f0330009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1312dc48aa2d1bac server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.398+0000 7f5ad82ea700 1 -- start start 2023-07-04T18:51:30.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4de060700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 0x7fc4d80c5bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e900 tx=0x7fc4d400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f033affd700 1 -- 172.21.15.77:0/2321370239 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330004230 con 0x7f033c0b1890 2023-07-04T18:51:30.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f033affd700 1 -- 172.21.15.77:0/2321370239 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0330004390 con 0x7f033c0b1890 2023-07-04T18:51:30.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f033affd700 1 -- 172.21.15.77:0/2321370239 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03300036a0 con 0x7f033c0b1890 2023-07-04T18:51:30.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400c850 con 0x7fc4d806c6f0 2023-07-04T18:51:30.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d400c9b0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400cd20 con 0x7fc4d806c6f0 2023-07-04T18:51:30.437 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d81ea4b0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.437 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d80c6130 con 0x7fc4d806c6f0 2023-07-04T18:51:30.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f5ad82ea700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad006c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f5ad82ea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ad00cdca0 0x7f5ad00ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f5ad82ea700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad006d890 con 0x7f5ad006c770 2023-07-04T18:51:30.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f5ad82ea700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad006cc80 con 0x7f5ad00cdca0 2023-07-04T18:51:30.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f5ad82ea700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad006cf60 con 0x7f5ad0159280 2023-07-04T18:51:30.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ad00cdca0 0x7f5ad00ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5ad00cdca0 0x7f5ad00ca010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44316/0 (socket says 172.21.15.77:44316) 2023-07-04T18:51:30.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 -- 172.21.15.77:0/3798608110 learned_addr learned my addr 172.21.15.77:0/3798608110 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.399+0000 7f79ec5ae700 1 -- 172.21.15.77:0/429463415 shutdown_connections 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 -- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 msgr2=0x7f033c1648e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 --2- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c1648e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0330009840 tx=0x7f0330009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 -- 172.21.15.77:0/2321370239 shutdown_connections 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 --2- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c1648e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 --2- 172.21.15.77:0/2321370239 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c157ac0 0x7f033c1643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 -- 172.21.15.77:0/2321370239 >> 172.21.15.77:0/2321370239 conn(0x7f033c069e40 msgr2=0x7f033c06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 -- 172.21.15.77:0/3798608110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 msgr2=0x7f5ad006c230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-04T18:51:30.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 --2- 172.21.15.77:0/3798608110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad006c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 -- 172.21.15.77:0/3798608110 >> v1:172.21.15.77:6789/0 conn(0x7f5ad006c770 legacy=0x7f5ad00cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 -- 172.21.15.77:0/3798608110 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad006d240 con 0x7f5ad00cdca0 2023-07-04T18:51:30.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f03424e2700 1 -- 172.21.15.77:0/2321370239 shutdown_connections 2023-07-04T18:51:30.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f5ad6887700 1 --2- 172.21.15.77:0/3798608110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad00ca010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0098f0 tx=0x7f5acc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506507a970214c8f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.400+0000 7f79ec5ae700 1 -- 172.21.15.77:0/429463415 wait complete. 2023-07-04T18:51:30.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.401+0000 7f03424e2700 1 -- 172.21.15.77:0/2321370239 wait complete. 2023-07-04T18:51:30.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.401+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d804e210 con 0x7fc4d806c6f0 2023-07-04T18:51:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.401+0000 7f03d1d9b700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 0x7f03b406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.402+0000 7f03d1d9b700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 0x7f03b406d1b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b47b0 tx=0x7f03c800f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.402+0000 7f79ec5ae700 1 Processor -- start 2023-07-04T18:51:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.402+0000 7f79ec5ae700 1 -- start start 2023-07-04T18:51:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f03424e2700 1 Processor -- start 2023-07-04T18:51:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f03424e2700 1 -- start start 2023-07-04T18:51:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.402+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc4d4013470 con 0x7fc4d806c6f0 2023-07-04T18:51:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7fc4ceffd700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 0x7fc4bc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc4d4089bd0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7fc4dd85f700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 0x7fc4bc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 0x7f79e40c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e41585e0 0x7f79e41f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40c3fc0 con 0x7f79e41585e0 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40b3200 con 0x7f79e40c3a70 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79ec5ae700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40b3410 con 0x7f79e40ced60 2023-07-04T18:51:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79eab4b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.403+0000 7f79eab4b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50132/0 (socket says 172.21.15.77:50132) 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79eab4b700 1 -- 172.21.15.77:0/3185963277 learned_addr learned my addr 172.21.15.77:0/3185963277 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 0x7f79e40c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 -- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 msgr2=0x7f79e4070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 -- 172.21.15.77:0/3185963277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e41585e0 msgr2=0x7f79e41f0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e41585e0 0x7f79e41f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 -- 172.21.15.77:0/3185963277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009de0 con 0x7f79e40ced60 2023-07-04T18:51:30.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79eab4b700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:30.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f5ad5084700 1 -- 172.21.15.77:0/3798608110 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00b8a0 con 0x7f5ad00cdca0 2023-07-04T18:51:30.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f5ad5084700 1 -- 172.21.15.77:0/3798608110 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc00ba00 con 0x7f5ad00cdca0 2023-07-04T18:51:30.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f5ad5084700 1 -- 172.21.15.77:0/3798608110 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc00bd90 con 0x7f5ad00cdca0 2023-07-04T18:51:30.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f79ea34a700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 0x7f79e40c1a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009fd0 tx=0x7f79d400b6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c1576f0 0x7f033c15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c151150 con 0x7f033c157ac0 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c158470 con 0x7f033c0b1890 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f03424e2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c158750 con 0x7f033c1576f0 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f0340a7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f0340a7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44330/0 (socket says 172.21.15.77:44330) 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.404+0000 7f0340a7f700 1 -- 172.21.15.77:0/84775045 learned_addr learned my addr 172.21.15.77:0/84775045 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 -- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c1576f0 msgr2=0x7f033c15a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c1576f0 0x7f033c15a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 -- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 msgr2=0x7f033c157e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c157e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 -- 172.21.15.77:0/84775045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03300094a0 con 0x7f033c157ac0 2023-07-04T18:51:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 -- 172.21.15.77:0/3798608110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 msgr2=0x7f5ad00ca010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/3798608110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad00ca010 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0098f0 tx=0x7f5acc009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:30.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a84019070 con 0x7f9a8c0cc270 2023-07-04T18:51:30.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f0340a7f700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0330004000 tx=0x7f03300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 -- 172.21.15.77:0/3798608110 shutdown_connections 2023-07-04T18:51:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/3798608110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad00ca010 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/3798608110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad006c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 -- 172.21.15.77:0/3798608110 >> 172.21.15.77:0/3798608110 conn(0x7f5ad006bba0 msgr2=0x7f5ad00b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 -- 172.21.15.77:0/3798608110 shutdown_connections 2023-07-04T18:51:30.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f5ad82ea700 1 -- 172.21.15.77:0/3798608110 wait complete. 2023-07-04T18:51:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f9a7affd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 0x7f9a6c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.405+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9a840aca40 con 0x7f9a8c0cc270 2023-07-04T18:51:30.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.406+0000 7fc4dd85f700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 0x7fc4bc06d1e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4d815b7f0 tx=0x7fc4c8001b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.406+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4010970 con 0x7f79e40ced60 2023-07-04T18:51:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.406+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4010af0 con 0x7f79e40ced60 2023-07-04T18:51:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.406+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4010ea0 con 0x7f79e40ced60 2023-07-04T18:51:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.407+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e40b36f0 con 0x7f79e40ced60 2023-07-04T18:51:30.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.407+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e40b3cb0 con 0x7f79e40ced60 2023-07-04T18:51:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.407+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f03c405db60 con 0x7f03d40ad700 2023-07-04T18:51:30.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.407+0000 7f5ad82ea700 1 Processor -- start 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.407+0000 7f5ad82ea700 1 -- start start 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.408+0000 7f5ad82ea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ad006c770 0x7f5ad00c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.408+0000 7f5ad82ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad82ea700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad014c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad82ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad014cdd0 con 0x7f5ad00cdca0 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad82ea700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad014d0b0 con 0x7f5ad006c770 2023-07-04T18:51:30.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad82ea700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00c4f50 con 0x7f5ad0159280 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44340/0 (socket says 172.21.15.77:44340) 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 -- 172.21.15.77:0/1727379243 learned_addr learned my addr 172.21.15.77:0/1727379243 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 -- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 msgr2=0x7f5ad014c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad014c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 -- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ad006c770 msgr2=0x7f5ad00c5a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ad006c770 0x7f5ad00c5a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 -- 172.21.15.77:0/1727379243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc009550 con 0x7f5ad00cdca0 2023-07-04T18:51:30.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f5ad5885700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800cc50 tx=0x7f5ac800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.406+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc4d4054df0 con 0x7fc4d806c6f0 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.409+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f79d4020360 con 0x7f79e40ced60 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.410+0000 7f79daffd700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 0x7f79c806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.410+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f79d408c9b0 con 0x7f79e40ced60 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.410+0000 7f79e9b49700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 0x7f79c806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.410+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e40b3ea0 con 0x7f79e40ced60 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.410+0000 7f79e9b49700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 0x7f79c806d1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79e4071de0 tx=0x7f79e0011000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.415+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f79d4057930 con 0x7f79e40ced60 2023-07-04T18:51:30.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.416+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800ade0 con 0x7f5ad00cdca0 2023-07-04T18:51:30.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.416+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac800ce50 con 0x7f5ad00cdca0 2023-07-04T18:51:30.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.416+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8003c20 con 0x7f5ad00cdca0 2023-07-04T18:51:30.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.416+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a84077b60 con 0x7f9a8c0cc270 2023-07-04T18:51:30.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.417+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad00c5230 con 0x7f5ad00cdca0 2023-07-04T18:51:30.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.417+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad014ca30 con 0x7f5ad00cdca0 2023-07-04T18:51:30.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.418+0000 7f9a8a59c700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 0x7f9a6c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.429+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330011630 con 0x7f033c157ac0 2023-07-04T18:51:30.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.429+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0330011790 con 0x7f033c157ac0 2023-07-04T18:51:30.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.429+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330011b40 con 0x7f033c157ac0 2023-07-04T18:51:30.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.420+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad00c5750 con 0x7f5ad00cdca0 2023-07-04T18:51:30.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.420+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5ac80050d0 con 0x7f5ad00cdca0 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.420+0000 7f5ac6ffd700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 0x7f5ab406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.420+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5ac8089620 con 0x7f5ad00cdca0 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.421+0000 7f5ad6086700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 0x7f5ab406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.421+0000 7f5ad6086700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 0x7f5ab406d190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0006fd0 tx=0x7f5ac0006bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.429+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c158a30 con 0x7f033c157ac0 2023-07-04T18:51:30.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.429+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c158ff0 con 0x7f033c157ac0 2023-07-04T18:51:30.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.431+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5ad00c5750 con 0x7f5ad00cdca0 2023-07-04T18:51:30.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.432+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c0052f0 con 0x7f033c157ac0 2023-07-04T18:51:30.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.433+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0330011ca0 con 0x7f033c157ac0 2023-07-04T18:51:30.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.433+0000 7f0338ff9700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 0x7f032806d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:30.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.433+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f033008acc0 con 0x7f033c157ac0 2023-07-04T18:51:30.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.440+0000 7f9a8a59c700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 0x7f9a6c06d260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c6cb0 tx=0x7f9a80009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.440+0000 7f033bfff700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 0x7f032806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:30.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.443+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0330055de0 con 0x7f033c157ac0 2023-07-04T18:51:30.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.444+0000 7f033bfff700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 0x7f032806d290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0334005a30 tx=0x7f0334004fd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:30.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.741+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f55180055a0 con 0x7f55380ce250 2023-07-04T18:51:30.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.741+0000 7f552dffb700 1 -- 172.21.15.77:0/4108227595 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5534025070 con 0x7f55380ce250 2023-07-04T18:51:30.750 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:30.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 msgr2=0x7f551c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 0x7f551c06d260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5538202c60 tx=0x7f5528008040 comp rx=0 tx=0).stop 2023-07-04T18:51:30.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 msgr2=0x7f55380b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f553806edb0 tx=0x7f5534004750 comp rx=0 tx=0).stop 2023-07-04T18:51:30.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc3d8005870 con 0x7fc3e00050b0 2023-07-04T18:51:30.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 shutdown_connections 2023-07-04T18:51:30.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f551c06adb0 0x7f551c06d260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f553815b240 0x7f55380707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55380ce250 0x7f55380b1740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 --2- 172.21.15.77:0/4108227595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55380b32a0 0x7f55380702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 >> 172.21.15.77:0/4108227595 conn(0x7f553806aad0 msgr2=0x7f55380b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.751+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 shutdown_connections 2023-07-04T18:51:30.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.751+0000 7f553f2f4700 1 -- 172.21.15.77:0/4108227595 wait complete. 2023-07-04T18:51:30.756 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:30.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.749+0000 7fc3df7fe700 1 -- 172.21.15.77:0/2275544221 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc3ec023030 con 0x7fc3e00050b0 2023-07-04T18:51:30.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.753+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 msgr2=0x7fc3e006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.753+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 0x7fc3e006eb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8157b60 tx=0x7fc3e8008040 comp rx=0 tx=0).stop 2023-07-04T18:51:30.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.753+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 msgr2=0x7fc3f80c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.753+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec011590 tx=0x7fc3ec011950 comp rx=0 tx=0).stop 2023-07-04T18:51:30.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 shutdown_connections 2023-07-04T18:51:30.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc3e006c690 0x7fc3e006eb40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3e00050b0 0x7fc3f80c6450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc3f80c44d0 0x7fc3f80c23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 --2- 172.21.15.77:0/2275544221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3f81e3550 0x7fc3f814fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.754+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 >> 172.21.15.77:0/2275544221 conn(0x7fc3f806aad0 msgr2=0x7fc3f80aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.755+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 shutdown_connections 2023-07-04T18:51:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.755+0000 7fc3fd634700 1 -- 172.21.15.77:0/2275544221 wait complete. 2023-07-04T18:51:30.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-07-04T18:51:30.772 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-07-04T18:51:30.776 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.794+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f79e404e210 con 0x7f79e40ced60 2023-07-04T18:51:30.796 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.795+0000 7f79daffd700 1 -- 172.21.15.77:0/3185963277 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f79d401d030 con 0x7f79e40ced60 2023-07-04T18:51:30.797 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:30.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 msgr2=0x7f79c806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 0x7f79c806d1c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79e4071de0 tx=0x7f79e0011000 comp rx=0 tx=0).stop 2023-07-04T18:51:30.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 msgr2=0x7f79e40c1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 0x7f79e40c1a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009fd0 tx=0x7f79d400b6c0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 shutdown_connections 2023-07-04T18:51:30.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f79c806ad10 0x7f79c806d1c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f79e40c3a70 0x7f79e4070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f79e40ced60 0x7f79e40c1a70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 --2- 172.21.15.77:0/3185963277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79e41585e0 0x7f79e41f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 >> 172.21.15.77:0/3185963277 conn(0x7f79e4069e40 msgr2=0x7f79e41565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 shutdown_connections 2023-07-04T18:51:30.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.797+0000 7f79ec5ae700 1 -- 172.21.15.77:0/3185963277 wait complete. 2023-07-04T18:51:30.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.799+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc4d81ea310 con 0x7fc4d806c6f0 2023-07-04T18:51:30.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.800+0000 7fc4ceffd700 1 -- 172.21.15.77:0/2380621648 <== mon.2 v2:172.21.15.82:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4d81ea310 con 0x7fc4d806c6f0 2023-07-04T18:51:30.803 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:30.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.803+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 msgr2=0x7fc4bc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.803+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 0x7fc4bc06d1e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4d815b7f0 tx=0x7fc4c8001b60 comp rx=0 tx=0).stop 2023-07-04T18:51:30.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.803+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 msgr2=0x7fc4d80c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.803+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 0x7fc4d80c5bf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400e900 tx=0x7fc4d400ecc0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 shutdown_connections 2023-07-04T18:51:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc4bc06ad30 0x7fc4bc06d1e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc4d81592d0 0x7fc4d8201a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc4d806c6f0 0x7fc4d80c5bf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 --2- 172.21.15.77:0/2380621648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4d80c8f00 0x7fc4d82014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 >> 172.21.15.77:0/2380621648 conn(0x7fc4d806aad0 msgr2=0x7fc4d80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 shutdown_connections 2023-07-04T18:51:30.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.804+0000 7fc4e02c4700 1 -- 172.21.15.77:0/2380621648 wait complete. 2023-07-04T18:51:30.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.807+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f03d40c3670 con 0x7f03d40ad700 2023-07-04T18:51:30.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.808+0000 7f03c2ffd700 1 -- 172.21.15.77:0/1024835126 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03c4061180 con 0x7f03d40ad700 2023-07-04T18:51:30.809 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:30.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.808+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f031c0055a0 con 0x7f033c157ac0 2023-07-04T18:51:30.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.808+0000 7f0338ff9700 1 -- 172.21.15.77:0/84775045 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f033001b440 con 0x7f033c157ac0 2023-07-04T18:51:30.811 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2023-07-04T18:51:30.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.810+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9a70005870 con 0x7f9a8c0cc270 2023-07-04T18:51:30.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 msgr2=0x7f032806d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 0x7f032806d290 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0334005a30 tx=0x7f0334004fd0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 msgr2=0x7f033c1502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0330004000 tx=0x7f03300047c0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f9a7affd700 1 -- 172.21.15.77:0/3734476055 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a8407b180 con 0x7f9a8c0cc270 2023-07-04T18:51:30.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 shutdown_connections 2023-07-04T18:51:30.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f032806ade0 0x7f032806d290 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f033c157ac0 0x7f033c1502e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f033c0b1890 0x7f033c157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 --2- 172.21.15.77:0/84775045 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f033c1576f0 0x7f033c15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 >> 172.21.15.77:0/84775045 conn(0x7f033c069e40 msgr2=0x7f033c163380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.817 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:30.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 shutdown_connections 2023-07-04T18:51:30.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.811+0000 7f03424e2700 1 -- 172.21.15.77:0/84775045 wait complete. 2023-07-04T18:51:30.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 msgr2=0x7f03b406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 0x7f03b406d1b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b47b0 tx=0x7f03c800f040 comp rx=0 tx=0).stop 2023-07-04T18:51:30.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 msgr2=0x7f03d4149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03c401a040 tx=0x7f03c4004750 comp rx=0 tx=0).stop 2023-07-04T18:51:30.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 msgr2=0x7f9a6c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 0x7f9a6c06d260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c0c6cb0 tx=0x7f9a80009190 comp rx=0 tx=0).stop 2023-07-04T18:51:30.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 msgr2=0x7f9a8c0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.813+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 0x7f9a8c0cc6e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c06ef60 tx=0x7f9a84004ad0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.822 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-07-04T18:51:30.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 shutdown_connections 2023-07-04T18:51:30.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9a6c06adb0 0x7f9a6c06d260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a8c0cc270 0x7f9a8c0cc6e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9a8c0cb980 0x7f9a8c0c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 --2- 172.21.15.77:0/3734476055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a8c0c6ec0 0x7f9a8c14beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 >> 172.21.15.77:0/3734476055 conn(0x7f9a8c06aad0 msgr2=0x7f9a8c0c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 shutdown_connections 2023-07-04T18:51:30.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f03b406ad00 0x7f03b406d1b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03d40ad700 0x7f03d4149c00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03d415be90 0x7f03d40c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 --2- 172.21.15.77:0/1024835126 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03d40cdb40 0x7f03d40c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 >> 172.21.15.77:0/1024835126 conn(0x7f03d406aad0 msgr2=0x7f03d40ae340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.825 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 shutdown_connections 2023-07-04T18:51:30.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.814+0000 7f9a90fdd700 1 -- 172.21.15.77:0/3734476055 wait complete. 2023-07-04T18:51:30.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.816+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 shutdown_connections 2023-07-04T18:51:30.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.816+0000 7f03d8bb7700 1 -- 172.21.15.77:0/1024835126 wait complete. 2023-07-04T18:51:30.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-07-04T18:51:30.828 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-07-04T18:51:30.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.827+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5ad00c5750 con 0x7f5ad00cdca0 2023-07-04T18:51:30.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.827+0000 7f5ac6ffd700 1 -- 172.21.15.77:0/1727379243 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ad00c5750 con 0x7f5ad00cdca0 2023-07-04T18:51:30.829 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.830 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2023-07-04T18:51:30.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 msgr2=0x7f5ab406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 0x7f5ab406d190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0006fd0 tx=0x7f5ac0006bc0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 msgr2=0x7f5ad014bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800cc50 tx=0x7f5ac800a340 comp rx=0 tx=0).stop 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 shutdown_connections 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5ab406ace0 0x7f5ab406d190 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5ad0159280 0x7f5ad014c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ad006c770 0x7f5ad00c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 --2- 172.21.15.77:0/1727379243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5ad00cdca0 0x7f5ad014bfb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:30.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 >> 172.21.15.77:0/1727379243 conn(0x7f5ad006bba0 msgr2=0x7f5ad00b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:30.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 shutdown_connections 2023-07-04T18:51:30.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:30.830+0000 7f5ad82ea700 1 -- 172.21.15.77:0/1727379243 wait complete. 2023-07-04T18:51:30.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-07-04T18:51:30.834 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2023-07-04T18:51:30.837 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2023-07-04T18:51:30.845 DEBUG:teuthology.parallel:result is None 2023-07-04T18:51:30.846 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-04T18:51:30.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:51:31.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.004+0000 7fd1ddab9700 1 Processor -- start 2023-07-04T18:51:31.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.004+0000 7fd1ddab9700 1 -- start start 2023-07-04T18:51:31.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1ddab9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd1d815c510 0x7fd1d8160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1ddab9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d80b0270 con 0x7fd1d806ef20 2023-07-04T18:51:31.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1ddab9700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d815c8e0 con 0x7fd1d80cf960 2023-07-04T18:51:31.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1ddab9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d8160f90 con 0x7fd1d815c510 2023-07-04T18:51:31.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd1d815c510 0x7fd1d8160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd1d806ef20 0x7fd1d80cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51490/0 (socket says 172.21.15.77:51490) 2023-07-04T18:51:31.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d6ffd700 1 -- 172.21.15.77:0/936514662 learned_addr learned my addr 172.21.15.77:0/936514662 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:31.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d77fe700 1 -- 172.21.15.77:0/936514662 >> v1:172.21.15.82:6789/0 conn(0x7fd1d80cf960 legacy=0x7fd1d80cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d77fe700 1 -- 172.21.15.77:0/936514662 >> v1:172.21.15.77:6789/0 conn(0x7fd1d806ef20 legacy=0x7fd1d80cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d77fe700 1 -- 172.21.15.77:0/936514662 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d80ce7e0 con 0x7fd1d815c510 2023-07-04T18:51:31.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d67fc700 1 -- 172.21.15.77:0/936514662 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657175217 0 0) 0x7fd1d80b0270 con 0x7fd1d806ef20 2023-07-04T18:51:31.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d77fe700 1 --2- 172.21.15.77:0/936514662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8160030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009840 tx=0x7fd1c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97993f993e0e22f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d67fc700 1 -- 172.21.15.77:0/936514662 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011830 con 0x7fd1d815c510 2023-07-04T18:51:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d67fc700 1 -- 172.21.15.77:0/936514662 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c8011990 con 0x7fd1d815c510 2023-07-04T18:51:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.005+0000 7fd1d67fc700 1 -- 172.21.15.77:0/936514662 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011d20 con 0x7fd1d815c510 2023-07-04T18:51:31.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 -- 172.21.15.77:0/936514662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 msgr2=0x7fd1d8160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/936514662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8160030 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8009840 tx=0x7fd1c8009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:31.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 -- 172.21.15.77:0/936514662 shutdown_connections 2023-07-04T18:51:31.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/936514662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8160030 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 -- 172.21.15.77:0/936514662 >> 172.21.15.77:0/936514662 conn(0x7fd1d806d7c0 msgr2=0x7fd1d806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:31.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 -- 172.21.15.77:0/936514662 shutdown_connections 2023-07-04T18:51:31.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.006+0000 7fd1ddab9700 1 -- 172.21.15.77:0/936514662 wait complete. 2023-07-04T18:51:31.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.007+0000 7fd1ddab9700 1 Processor -- start 2023-07-04T18:51:31.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.007+0000 7fd1ddab9700 1 -- start start 2023-07-04T18:51:31.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1d806ef20 0x7fd1d80b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 0x7fd1d8152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80c92d0 con 0x7fd1d815c510 2023-07-04T18:51:31.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80c8170 con 0x7fd1d806ef20 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1ddab9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80c8450 con 0x7fd1d80cf960 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44360/0 (socket says 172.21.15.77:44360) 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 -- 172.21.15.77:0/2585993246 learned_addr learned my addr 172.21.15.77:0/2585993246 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d77fe700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1d806ef20 0x7fd1d80b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d6ffd700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 0x7fd1d8152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 -- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 msgr2=0x7fd1d8152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 0x7fd1d8152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 -- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1d806ef20 msgr2=0x7fd1d80b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1d806ef20 0x7fd1d80b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d7fff700 1 -- 172.21.15.77:0/2585993246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c80094a0 con 0x7fd1d815c510 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.008+0000 7fd1d6ffd700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 0x7fd1d8152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:31.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1d7fff700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000cb50 tx=0x7fd1d000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0003c00 con 0x7fd1d815c510 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d0003d60 con 0x7fd1d815c510 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d000aac0 con 0x7fd1d815c510 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80c8730 con 0x7fd1d815c510 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.009+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d8154220 con 0x7fd1d815c510 2023-07-04T18:51:31.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.010+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d804e210 con 0x7fd1d815c510 2023-07-04T18:51:31.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.010+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd1d000ac20 con 0x7fd1d815c510 2023-07-04T18:51:31.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.011+0000 7fd1bffff700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.011+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd1d00893b0 con 0x7fd1d815c510 2023-07-04T18:51:31.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.011+0000 7fd1d77fe700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.011+0000 7fd1d77fe700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 0x7fd1c406d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80090b0 tx=0x7fd1c801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.015+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd1d00544f0 con 0x7fd1d815c510 2023-07-04T18:51:31.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.197+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1d81543b0 con 0x7fd1c406ace0 2023-07-04T18:51:31.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.205+0000 7fd1bffff700 1 -- 172.21.15.77:0/2585993246 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286620 (secure 0 0 0) 0x7fd1d81543b0 con 0x7fd1c406ace0 2023-07-04T18:51:31.210 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:31.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 msgr2=0x7fd1c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 0x7fd1c406d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80090b0 tx=0x7fd1c801c040 comp rx=0 tx=0).stop 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 msgr2=0x7fd1d8153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1d000cb50 tx=0x7fd1d000ce60 comp rx=0 tx=0).stop 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 shutdown_connections 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd1c406ace0 0x7fd1c406d190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1d815c510 0x7fd1d8153490 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1d806ef20 0x7fd1d80b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 --2- 172.21.15.77:0/2585993246 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd1d80cf960 0x7fd1d8152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 >> 172.21.15.77:0/2585993246 conn(0x7fd1d806d7c0 msgr2=0x7fd1d80cc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 shutdown_connections 2023-07-04T18:51:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.209+0000 7fd1ddab9700 1 -- 172.21.15.77:0/2585993246 wait complete. 2023-07-04T18:51:31.214 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:51:31.223 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-07-04T18:51:31.178587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218728,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755992,"statfs":{"total":773094113280,"available":772870135808,"internally_reserved":0,"allocated":3186688,"data_stored":1902252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.875834"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773578+0000","last_change":"2023-07-04T18:51:22.314047+0000","last_active":"2023-07-04T18:51:24.773578+0000","last_peered":"2023-07-04T18:51:24.773578+0000","last_clean":"2023-07-04T18:51:24.773578+0000","last_became_active":"2023-07-04T18:51:22.313772+0000","last_became_peered":"2023-07-04T18:51:22.313772+0000","last_unstale":"2023-07-04T18:51:24.773578+0000","last_undegraded":"2023-07-04T18:51:24.773578+0000","last_fullsized":"2023-07-04T18:51:24.773578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774630+0000","last_change":"2023-07-04T18:51:22.324407+0000","last_active":"2023-07-04T18:51:24.774630+0000","last_peered":"2023-07-04T18:51:24.774630+0000","last_clean":"2023-07-04T18:51:24.774630+0000","last_became_active":"2023-07-04T18:51:22.323862+0000","last_became_peered":"2023-07-04T18:51:22.323862+0000","last_unstale":"2023-07-04T18:51:24.774630+0000","last_undegraded":"2023-07-04T18:51:24.774630+0000","last_fullsized":"2023-07-04T18:51:24.774630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304530+0000","last_change":"2023-07-04T18:51:22.319081+0000","last_active":"2023-07-04T18:51:23.304530+0000","last_peered":"2023-07-04T18:51:23.304530+0000","last_clean":"2023-07-04T18:51:23.304530+0000","last_became_active":"2023-07-04T18:51:22.318806+0000","last_became_peered":"2023-07-04T18:51:22.318806+0000","last_unstale":"2023-07-04T18:51:23.304530+0000","last_undegraded":"2023-07-04T18:51:23.304530+0000","last_fullsized":"2023-07-04T18:51:23.304530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303285+0000","last_change":"2023-07-04T18:51:22.319728+0000","last_active":"2023-07-04T18:51:23.303285+0000","last_peered":"2023-07-04T18:51:23.303285+0000","last_clean":"2023-07-04T18:51:23.303285+0000","last_became_active":"2023-07-04T18:51:22.319417+0000","last_became_peered":"2023-07-04T18:51:22.319417+0000","last_unstale":"2023-07-04T18:51:23.303285+0000","last_undegraded":"2023-07-04T18:51:23.303285+0000","last_fullsized":"2023-07-04T18:51:23.303285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752506+0000","last_change":"2023-07-04T18:51:22.317945+0000","last_active":"2023-07-04T18:51:24.752506+0000","last_peered":"2023-07-04T18:51:24.752506+0000","last_clean":"2023-07-04T18:51:24.752506+0000","last_became_active":"2023-07-04T18:51:22.317635+0000","last_became_peered":"2023-07-04T18:51:22.317635+0000","last_unstale":"2023-07-04T18:51:24.752506+0000","last_undegraded":"2023-07-04T18:51:24.752506+0000","last_fullsized":"2023-07-04T18:51:24.752506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304259+0000","last_change":"2023-07-04T18:51:22.320918+0000","last_active":"2023-07-04T18:51:23.304259+0000","last_peered":"2023-07-04T18:51:23.304259+0000","last_clean":"2023-07-04T18:51:23.304259+0000","last_became_active":"2023-07-04T18:51:22.320502+0000","last_became_peered":"2023-07-04T18:51:22.320502+0000","last_unstale":"2023-07-04T18:51:23.304259+0000","last_undegraded":"2023-07-04T18:51:23.304259+0000","last_fullsized":"2023-07-04T18:51:23.304259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450875+0000","last_change":"2023-07-04T18:51:22.323607+0000","last_active":"2023-07-04T18:51:23.450875+0000","last_peered":"2023-07-04T18:51:23.450875+0000","last_clean":"2023-07-04T18:51:23.450875+0000","last_became_active":"2023-07-04T18:51:22.322940+0000","last_became_peered":"2023-07-04T18:51:22.322940+0000","last_unstale":"2023-07-04T18:51:23.450875+0000","last_undegraded":"2023-07-04T18:51:23.450875+0000","last_fullsized":"2023-07-04T18:51:23.450875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774287+0000","last_change":"2023-07-04T18:51:22.321665+0000","last_active":"2023-07-04T18:51:24.774287+0000","last_peered":"2023-07-04T18:51:24.774287+0000","last_clean":"2023-07-04T18:51:24.774287+0000","last_became_active":"2023-07-04T18:51:22.321426+0000","last_became_peered":"2023-07-04T18:51:22.321426+0000","last_unstale":"2023-07-04T18:51:24.774287+0000","last_undegraded":"2023-07-04T18:51:24.774287+0000","last_fullsized":"2023-07-04T18:51:24.774287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752371+0000","last_change":"2023-07-04T18:51:22.318535+0000","last_active":"2023-07-04T18:51:24.752371+0000","last_peered":"2023-07-04T18:51:24.752371+0000","last_clean":"2023-07-04T18:51:24.752371+0000","last_became_active":"2023-07-04T18:51:22.318160+0000","last_became_peered":"2023-07-04T18:51:22.318160+0000","last_unstale":"2023-07-04T18:51:24.752371+0000","last_undegraded":"2023-07-04T18:51:24.752371+0000","last_fullsized":"2023-07-04T18:51:24.752371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774802+0000","last_change":"2023-07-04T18:51:22.319854+0000","last_active":"2023-07-04T18:51:24.774802+0000","last_peered":"2023-07-04T18:51:24.774802+0000","last_clean":"2023-07-04T18:51:24.774802+0000","last_became_active":"2023-07-04T18:51:22.319581+0000","last_became_peered":"2023-07-04T18:51:22.319581+0000","last_unstale":"2023-07-04T18:51:24.774802+0000","last_undegraded":"2023-07-04T18:51:24.774802+0000","last_fullsized":"2023-07-04T18:51:24.774802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:28.243222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304943+0000","last_change":"2023-07-04T18:51:22.322222+0000","last_active":"2023-07-04T18:51:23.304943+0000","last_peered":"2023-07-04T18:51:23.304943+0000","last_clean":"2023-07-04T18:51:23.304943+0000","last_became_active":"2023-07-04T18:51:22.321944+0000","last_became_peered":"2023-07-04T18:51:22.321944+0000","last_unstale":"2023-07-04T18:51:23.304943+0000","last_undegraded":"2023-07-04T18:51:23.304943+0000","last_fullsized":"2023-07-04T18:51:23.304943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:53.251606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773693+0000","last_change":"2023-07-04T18:51:22.318773+0000","last_active":"2023-07-04T18:51:24.773693+0000","last_peered":"2023-07-04T18:51:24.773693+0000","last_clean":"2023-07-04T18:51:24.773693+0000","last_became_active":"2023-07-04T18:51:22.318495+0000","last_became_peered":"2023-07-04T18:51:22.318495+0000","last_unstale":"2023-07-04T18:51:24.773693+0000","last_undegraded":"2023-07-04T18:51:24.773693+0000","last_fullsized":"2023-07-04T18:51:24.773693+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773431+0000","last_change":"2023-07-04T18:51:22.313313+0000","last_active":"2023-07-04T18:51:24.773431+0000","last_peered":"2023-07-04T18:51:24.773431+0000","last_clean":"2023-07-04T18:51:24.773431+0000","last_became_active":"2023-07-04T18:51:22.313049+0000","last_became_peered":"2023-07-04T18:51:22.313049+0000","last_unstale":"2023-07-04T18:51:24.773431+0000","last_undegraded":"2023-07-04T18:51:24.773431+0000","last_fullsized":"2023-07-04T18:51:24.773431+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:18.636942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774464+0000","last_change":"2023-07-04T18:51:22.322812+0000","last_active":"2023-07-04T18:51:24.774464+0000","last_peered":"2023-07-04T18:51:24.774464+0000","last_clean":"2023-07-04T18:51:24.774464+0000","last_became_active":"2023-07-04T18:51:22.318629+0000","last_became_peered":"2023-07-04T18:51:22.318629+0000","last_unstale":"2023-07-04T18:51:24.774464+0000","last_undegraded":"2023-07-04T18:51:24.774464+0000","last_fullsized":"2023-07-04T18:51:24.774464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450996+0000","last_change":"2023-07-04T18:51:22.323740+0000","last_active":"2023-07-04T18:51:23.450996+0000","last_peered":"2023-07-04T18:51:23.450996+0000","last_clean":"2023-07-04T18:51:23.450996+0000","last_became_active":"2023-07-04T18:51:22.323321+0000","last_became_peered":"2023-07-04T18:51:22.323321+0000","last_unstale":"2023-07-04T18:51:23.450996+0000","last_undegraded":"2023-07-04T18:51:23.450996+0000","last_fullsized":"2023-07-04T18:51:23.450996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:33:48.187398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304015+0000","last_change":"2023-07-04T18:51:22.318780+0000","last_active":"2023-07-04T18:51:23.304015+0000","last_peered":"2023-07-04T18:51:23.304015+0000","last_clean":"2023-07-04T18:51:23.304015+0000","last_became_active":"2023-07-04T18:51:22.318460+0000","last_became_peered":"2023-07-04T18:51:22.318460+0000","last_unstale":"2023-07-04T18:51:23.304015+0000","last_undegraded":"2023-07-04T18:51:23.304015+0000","last_fullsized":"2023-07-04T18:51:23.304015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752520+0000","last_change":"2023-07-04T18:51:22.318170+0000","last_active":"2023-07-04T18:51:24.752520+0000","last_peered":"2023-07-04T18:51:24.752520+0000","last_clean":"2023-07-04T18:51:24.752520+0000","last_became_active":"2023-07-04T18:51:22.317879+0000","last_became_peered":"2023-07-04T18:51:22.317879+0000","last_unstale":"2023-07-04T18:51:24.752520+0000","last_undegraded":"2023-07-04T18:51:24.752520+0000","last_fullsized":"2023-07-04T18:51:24.752520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304493+0000","last_change":"2023-07-04T18:51:22.318559+0000","last_active":"2023-07-04T18:51:23.304493+0000","last_peered":"2023-07-04T18:51:23.304493+0000","last_clean":"2023-07-04T18:51:23.304493+0000","last_became_active":"2023-07-04T18:51:22.317739+0000","last_became_peered":"2023-07-04T18:51:22.317739+0000","last_unstale":"2023-07-04T18:51:23.304493+0000","last_undegraded":"2023-07-04T18:51:23.304493+0000","last_fullsized":"2023-07-04T18:51:23.304493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773617+0000","last_change":"2023-07-04T18:51:22.310959+0000","last_active":"2023-07-04T18:51:24.773617+0000","last_peered":"2023-07-04T18:51:24.773617+0000","last_clean":"2023-07-04T18:51:24.773617+0000","last_became_active":"2023-07-04T18:51:22.310668+0000","last_became_peered":"2023-07-04T18:51:22.310668+0000","last_unstale":"2023-07-04T18:51:24.773617+0000","last_undegraded":"2023-07-04T18:51:24.773617+0000","last_fullsized":"2023-07-04T18:51:24.773617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304398+0000","last_change":"2023-07-04T18:51:22.318672+0000","last_active":"2023-07-04T18:51:23.304398+0000","last_peered":"2023-07-04T18:51:23.304398+0000","last_clean":"2023-07-04T18:51:23.304398+0000","last_became_active":"2023-07-04T18:51:22.317856+0000","last_became_peered":"2023-07-04T18:51:22.317856+0000","last_unstale":"2023-07-04T18:51:23.304398+0000","last_undegraded":"2023-07-04T18:51:23.304398+0000","last_fullsized":"2023-07-04T18:51:23.304398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451144+0000","last_change":"2023-07-04T18:51:22.321788+0000","last_active":"2023-07-04T18:51:23.451144+0000","last_peered":"2023-07-04T18:51:23.451144+0000","last_clean":"2023-07-04T18:51:23.451144+0000","last_became_active":"2023-07-04T18:51:22.321394+0000","last_became_peered":"2023-07-04T18:51:22.321394+0000","last_unstale":"2023-07-04T18:51:23.451144+0000","last_undegraded":"2023-07-04T18:51:23.451144+0000","last_fullsized":"2023-07-04T18:51:23.451144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:00:25.437637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752367+0000","last_change":"2023-07-04T18:51:22.321742+0000","last_active":"2023-07-04T18:51:24.752367+0000","last_peered":"2023-07-04T18:51:24.752367+0000","last_clean":"2023-07-04T18:51:24.752367+0000","last_became_active":"2023-07-04T18:51:22.321526+0000","last_became_peered":"2023-07-04T18:51:22.321526+0000","last_unstale":"2023-07-04T18:51:24.752367+0000","last_undegraded":"2023-07-04T18:51:24.752367+0000","last_fullsized":"2023-07-04T18:51:24.752367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303613+0000","last_change":"2023-07-04T18:51:22.321095+0000","last_active":"2023-07-04T18:51:23.303613+0000","last_peered":"2023-07-04T18:51:23.303613+0000","last_clean":"2023-07-04T18:51:23.303613+0000","last_became_active":"2023-07-04T18:51:22.320848+0000","last_became_peered":"2023-07-04T18:51:22.320848+0000","last_unstale":"2023-07-04T18:51:23.303613+0000","last_undegraded":"2023-07-04T18:51:23.303613+0000","last_fullsized":"2023-07-04T18:51:23.303613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450972+0000","last_change":"2023-07-04T18:51:22.322035+0000","last_active":"2023-07-04T18:51:23.450972+0000","last_peered":"2023-07-04T18:51:23.450972+0000","last_clean":"2023-07-04T18:51:23.450972+0000","last_became_active":"2023-07-04T18:51:22.321725+0000","last_became_peered":"2023-07-04T18:51:22.321725+0000","last_unstale":"2023-07-04T18:51:23.450972+0000","last_undegraded":"2023-07-04T18:51:23.450972+0000","last_fullsized":"2023-07-04T18:51:23.450972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417751+0000","last_change":"2023-07-04T18:51:22.311616+0000","last_active":"2023-07-04T18:51:23.417751+0000","last_peered":"2023-07-04T18:51:23.417751+0000","last_clean":"2023-07-04T18:51:23.417751+0000","last_became_active":"2023-07-04T18:51:22.311344+0000","last_became_peered":"2023-07-04T18:51:22.311344+0000","last_unstale":"2023-07-04T18:51:23.417751+0000","last_undegraded":"2023-07-04T18:51:23.417751+0000","last_fullsized":"2023-07-04T18:51:23.417751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417648+0000","last_change":"2023-07-04T18:51:22.311562+0000","last_active":"2023-07-04T18:51:23.417648+0000","last_peered":"2023-07-04T18:51:23.417648+0000","last_clean":"2023-07-04T18:51:23.417648+0000","last_became_active":"2023-07-04T18:51:22.311234+0000","last_became_peered":"2023-07-04T18:51:22.311234+0000","last_unstale":"2023-07-04T18:51:23.417648+0000","last_undegraded":"2023-07-04T18:51:23.417648+0000","last_fullsized":"2023-07-04T18:51:23.417648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774505+0000","last_change":"2023-07-04T18:51:22.325120+0000","last_active":"2023-07-04T18:51:24.774505+0000","last_peered":"2023-07-04T18:51:24.774505+0000","last_clean":"2023-07-04T18:51:24.774505+0000","last_became_active":"2023-07-04T18:51:22.324834+0000","last_became_peered":"2023-07-04T18:51:22.324834+0000","last_unstale":"2023-07-04T18:51:24.774505+0000","last_undegraded":"2023-07-04T18:51:24.774505+0000","last_fullsized":"2023-07-04T18:51:24.774505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304437+0000","last_change":"2023-07-04T18:51:21.460823+0000","last_active":"2023-07-04T18:51:23.304437+0000","last_peered":"2023-07-04T18:51:23.304437+0000","last_clean":"2023-07-04T18:51:23.304437+0000","last_became_active":"2023-07-04T18:51:21.460492+0000","last_became_peered":"2023-07-04T18:51:21.460492+0000","last_unstale":"2023-07-04T18:51:23.304437+0000","last_undegraded":"2023-07-04T18:51:23.304437+0000","last_fullsized":"2023-07-04T18:51:23.304437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303755+0000","last_change":"2023-07-04T18:51:22.313093+0000","last_active":"2023-07-04T18:51:23.303755+0000","last_peered":"2023-07-04T18:51:23.303755+0000","last_clean":"2023-07-04T18:51:23.303755+0000","last_became_active":"2023-07-04T18:51:22.312832+0000","last_became_peered":"2023-07-04T18:51:22.312832+0000","last_unstale":"2023-07-04T18:51:23.303755+0000","last_undegraded":"2023-07-04T18:51:23.303755+0000","last_fullsized":"2023-07-04T18:51:23.303755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417409+0000","last_change":"2023-07-04T18:51:21.471442+0000","last_active":"2023-07-04T18:51:23.417409+0000","last_peered":"2023-07-04T18:51:23.417409+0000","last_clean":"2023-07-04T18:51:23.417409+0000","last_became_active":"2023-07-04T18:51:21.471130+0000","last_became_peered":"2023-07-04T18:51:21.471130+0000","last_unstale":"2023-07-04T18:51:23.417409+0000","last_undegraded":"2023-07-04T18:51:23.417409+0000","last_fullsized":"2023-07-04T18:51:23.417409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:38.465858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752486+0000","last_change":"2023-07-04T18:51:22.322483+0000","last_active":"2023-07-04T18:51:24.752486+0000","last_peered":"2023-07-04T18:51:24.752486+0000","last_clean":"2023-07-04T18:51:24.752486+0000","last_became_active":"2023-07-04T18:51:22.322262+0000","last_became_peered":"2023-07-04T18:51:22.322262+0000","last_unstale":"2023-07-04T18:51:24.752486+0000","last_undegraded":"2023-07-04T18:51:24.752486+0000","last_fullsized":"2023-07-04T18:51:24.752486+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303402+0000","last_change":"2023-07-04T18:51:21.470525+0000","last_active":"2023-07-04T18:51:23.303402+0000","last_peered":"2023-07-04T18:51:23.303402+0000","last_clean":"2023-07-04T18:51:23.303402+0000","last_became_active":"2023-07-04T18:51:21.470230+0000","last_became_peered":"2023-07-04T18:51:21.470230+0000","last_unstale":"2023-07-04T18:51:23.303402+0000","last_undegraded":"2023-07-04T18:51:23.303402+0000","last_fullsized":"2023-07-04T18:51:23.303402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303729+0000","last_change":"2023-07-04T18:51:22.323982+0000","last_active":"2023-07-04T18:51:23.303729+0000","last_peered":"2023-07-04T18:51:23.303729+0000","last_clean":"2023-07-04T18:51:23.303729+0000","last_became_active":"2023-07-04T18:51:22.323650+0000","last_became_peered":"2023-07-04T18:51:22.323650+0000","last_unstale":"2023-07-04T18:51:23.303729+0000","last_undegraded":"2023-07-04T18:51:23.303729+0000","last_fullsized":"2023-07-04T18:51:23.303729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752611+0000","last_change":"2023-07-04T18:51:21.464672+0000","last_active":"2023-07-04T18:51:24.752611+0000","last_peered":"2023-07-04T18:51:24.752611+0000","last_clean":"2023-07-04T18:51:24.752611+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:24.752611+0000","last_undegraded":"2023-07-04T18:51:24.752611+0000","last_fullsized":"2023-07-04T18:51:24.752611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:09.337120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303369+0000","last_change":"2023-07-04T18:51:22.311834+0000","last_active":"2023-07-04T18:51:23.303369+0000","last_peered":"2023-07-04T18:51:23.303369+0000","last_clean":"2023-07-04T18:51:23.303369+0000","last_became_active":"2023-07-04T18:51:22.311536+0000","last_became_peered":"2023-07-04T18:51:22.311536+0000","last_unstale":"2023-07-04T18:51:23.303369+0000","last_undegraded":"2023-07-04T18:51:23.303369+0000","last_fullsized":"2023-07-04T18:51:23.303369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774759+0000","last_change":"2023-07-04T18:51:21.464692+0000","last_active":"2023-07-04T18:51:24.774759+0000","last_peered":"2023-07-04T18:51:24.774759+0000","last_clean":"2023-07-04T18:51:24.774759+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:24.774759+0000","last_undegraded":"2023-07-04T18:51:24.774759+0000","last_fullsized":"2023-07-04T18:51:24.774759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:52.249980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303533+0000","last_change":"2023-07-04T18:51:22.320250+0000","last_active":"2023-07-04T18:51:23.303533+0000","last_peered":"2023-07-04T18:51:23.303533+0000","last_clean":"2023-07-04T18:51:23.303533+0000","last_became_active":"2023-07-04T18:51:22.319730+0000","last_became_peered":"2023-07-04T18:51:22.319730+0000","last_unstale":"2023-07-04T18:51:23.303533+0000","last_undegraded":"2023-07-04T18:51:23.303533+0000","last_fullsized":"2023-07-04T18:51:23.303533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451104+0000","last_change":"2023-07-04T18:51:21.471723+0000","last_active":"2023-07-04T18:51:23.451104+0000","last_peered":"2023-07-04T18:51:23.451104+0000","last_clean":"2023-07-04T18:51:23.451104+0000","last_became_active":"2023-07-04T18:51:21.471500+0000","last_became_peered":"2023-07-04T18:51:21.471500+0000","last_unstale":"2023-07-04T18:51:23.451104+0000","last_undegraded":"2023-07-04T18:51:23.451104+0000","last_fullsized":"2023-07-04T18:51:23.451104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:14:51.732204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417449+0000","last_change":"2023-07-04T18:51:22.311576+0000","last_active":"2023-07-04T18:51:23.417449+0000","last_peered":"2023-07-04T18:51:23.417449+0000","last_clean":"2023-07-04T18:51:23.417449+0000","last_became_active":"2023-07-04T18:51:22.311212+0000","last_became_peered":"2023-07-04T18:51:22.311212+0000","last_unstale":"2023-07-04T18:51:23.417449+0000","last_undegraded":"2023-07-04T18:51:23.417449+0000","last_fullsized":"2023-07-04T18:51:23.417449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304622+0000","last_change":"2023-07-04T18:51:21.464597+0000","last_active":"2023-07-04T18:51:23.304622+0000","last_peered":"2023-07-04T18:51:23.304622+0000","last_clean":"2023-07-04T18:51:23.304622+0000","last_became_active":"2023-07-04T18:51:21.464263+0000","last_became_peered":"2023-07-04T18:51:21.464263+0000","last_unstale":"2023-07-04T18:51:23.304622+0000","last_undegraded":"2023-07-04T18:51:23.304622+0000","last_fullsized":"2023-07-04T18:51:23.304622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304535+0000","last_change":"2023-07-04T18:51:22.318474+0000","last_active":"2023-07-04T18:51:23.304535+0000","last_peered":"2023-07-04T18:51:23.304535+0000","last_clean":"2023-07-04T18:51:23.304535+0000","last_became_active":"2023-07-04T18:51:22.317476+0000","last_became_peered":"2023-07-04T18:51:22.317476+0000","last_unstale":"2023-07-04T18:51:23.304535+0000","last_undegraded":"2023-07-04T18:51:23.304535+0000","last_fullsized":"2023-07-04T18:51:23.304535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450969+0000","last_change":"2023-07-04T18:51:21.467028+0000","last_active":"2023-07-04T18:51:23.450969+0000","last_peered":"2023-07-04T18:51:23.450969+0000","last_clean":"2023-07-04T18:51:23.450969+0000","last_became_active":"2023-07-04T18:51:21.466697+0000","last_became_peered":"2023-07-04T18:51:21.466697+0000","last_unstale":"2023-07-04T18:51:23.450969+0000","last_undegraded":"2023-07-04T18:51:23.450969+0000","last_fullsized":"2023-07-04T18:51:23.450969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303917+0000","last_change":"2023-07-04T18:51:22.320412+0000","last_active":"2023-07-04T18:51:23.303917+0000","last_peered":"2023-07-04T18:51:23.303917+0000","last_clean":"2023-07-04T18:51:23.303917+0000","last_became_active":"2023-07-04T18:51:22.320022+0000","last_became_peered":"2023-07-04T18:51:22.320022+0000","last_unstale":"2023-07-04T18:51:23.303917+0000","last_undegraded":"2023-07-04T18:51:23.303917+0000","last_fullsized":"2023-07-04T18:51:23.303917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752206+0000","last_change":"2023-07-04T18:51:21.465044+0000","last_active":"2023-07-04T18:51:24.752206+0000","last_peered":"2023-07-04T18:51:24.752206+0000","last_clean":"2023-07-04T18:51:24.752206+0000","last_became_active":"2023-07-04T18:51:21.464683+0000","last_became_peered":"2023-07-04T18:51:21.464683+0000","last_unstale":"2023-07-04T18:51:24.752206+0000","last_undegraded":"2023-07-04T18:51:24.752206+0000","last_fullsized":"2023-07-04T18:51:24.752206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:09.767759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774581+0000","last_change":"2023-07-04T18:51:22.324545+0000","last_active":"2023-07-04T18:51:24.774581+0000","last_peered":"2023-07-04T18:51:24.774581+0000","last_clean":"2023-07-04T18:51:24.774581+0000","last_became_active":"2023-07-04T18:51:22.324168+0000","last_became_peered":"2023-07-04T18:51:22.324168+0000","last_unstale":"2023-07-04T18:51:24.774581+0000","last_undegraded":"2023-07-04T18:51:24.774581+0000","last_fullsized":"2023-07-04T18:51:24.774581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:21.786665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450810+0000","last_change":"2023-07-04T18:51:21.468878+0000","last_active":"2023-07-04T18:51:23.450810+0000","last_peered":"2023-07-04T18:51:23.450810+0000","last_clean":"2023-07-04T18:51:23.450810+0000","last_became_active":"2023-07-04T18:51:21.468229+0000","last_became_peered":"2023-07-04T18:51:21.468229+0000","last_unstale":"2023-07-04T18:51:23.450810+0000","last_undegraded":"2023-07-04T18:51:23.450810+0000","last_fullsized":"2023-07-04T18:51:23.450810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752369+0000","last_change":"2023-07-04T18:51:22.317975+0000","last_active":"2023-07-04T18:51:24.752369+0000","last_peered":"2023-07-04T18:51:24.752369+0000","last_clean":"2023-07-04T18:51:24.752369+0000","last_became_active":"2023-07-04T18:51:22.317731+0000","last_became_peered":"2023-07-04T18:51:22.317731+0000","last_unstale":"2023-07-04T18:51:24.752369+0000","last_undegraded":"2023-07-04T18:51:24.752369+0000","last_fullsized":"2023-07-04T18:51:24.752369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303059+0000","last_change":"2023-07-04T18:51:21.460954+0000","last_active":"2023-07-04T18:51:23.303059+0000","last_peered":"2023-07-04T18:51:23.303059+0000","last_clean":"2023-07-04T18:51:23.303059+0000","last_became_active":"2023-07-04T18:51:21.460663+0000","last_became_peered":"2023-07-04T18:51:21.460663+0000","last_unstale":"2023-07-04T18:51:23.303059+0000","last_undegraded":"2023-07-04T18:51:23.303059+0000","last_fullsized":"2023-07-04T18:51:23.303059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:08.443319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304053+0000","last_change":"2023-07-04T18:51:22.313798+0000","last_active":"2023-07-04T18:51:23.304053+0000","last_peered":"2023-07-04T18:51:23.304053+0000","last_clean":"2023-07-04T18:51:23.304053+0000","last_became_active":"2023-07-04T18:51:22.313526+0000","last_became_peered":"2023-07-04T18:51:22.313526+0000","last_unstale":"2023-07-04T18:51:23.304053+0000","last_undegraded":"2023-07-04T18:51:23.304053+0000","last_fullsized":"2023-07-04T18:51:23.304053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451341+0000","last_change":"2023-07-04T18:51:21.468130+0000","last_active":"2023-07-04T18:51:23.451341+0000","last_peered":"2023-07-04T18:51:23.451341+0000","last_clean":"2023-07-04T18:51:23.451341+0000","last_became_active":"2023-07-04T18:51:21.467832+0000","last_became_peered":"2023-07-04T18:51:21.467832+0000","last_unstale":"2023-07-04T18:51:23.451341+0000","last_undegraded":"2023-07-04T18:51:23.451341+0000","last_fullsized":"2023-07-04T18:51:23.451341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:02:58.666739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774284+0000","last_change":"2023-07-04T18:51:22.325399+0000","last_active":"2023-07-04T18:51:24.774284+0000","last_peered":"2023-07-04T18:51:24.774284+0000","last_clean":"2023-07-04T18:51:24.774284+0000","last_became_active":"2023-07-04T18:51:22.324669+0000","last_became_peered":"2023-07-04T18:51:22.324669+0000","last_unstale":"2023-07-04T18:51:24.774284+0000","last_undegraded":"2023-07-04T18:51:24.774284+0000","last_fullsized":"2023-07-04T18:51:24.774284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451202+0000","last_change":"2023-07-04T18:51:21.466954+0000","last_active":"2023-07-04T18:51:23.451202+0000","last_peered":"2023-07-04T18:51:23.451202+0000","last_clean":"2023-07-04T18:51:23.451202+0000","last_became_active":"2023-07-04T18:51:21.466625+0000","last_became_peered":"2023-07-04T18:51:21.466625+0000","last_unstale":"2023-07-04T18:51:23.451202+0000","last_undegraded":"2023-07-04T18:51:23.451202+0000","last_fullsized":"2023-07-04T18:51:23.451202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:17:35.188150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304676+0000","last_change":"2023-07-04T18:51:22.321314+0000","last_active":"2023-07-04T18:51:23.304676+0000","last_peered":"2023-07-04T18:51:23.304676+0000","last_clean":"2023-07-04T18:51:23.304676+0000","last_became_active":"2023-07-04T18:51:22.321008+0000","last_became_peered":"2023-07-04T18:51:22.321008+0000","last_unstale":"2023-07-04T18:51:23.304676+0000","last_undegraded":"2023-07-04T18:51:23.304676+0000","last_fullsized":"2023-07-04T18:51:23.304676+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451276+0000","last_change":"2023-07-04T18:51:21.468101+0000","last_active":"2023-07-04T18:51:23.451276+0000","last_peered":"2023-07-04T18:51:23.451276+0000","last_clean":"2023-07-04T18:51:23.451276+0000","last_became_active":"2023-07-04T18:51:21.467813+0000","last_became_peered":"2023-07-04T18:51:21.467813+0000","last_unstale":"2023-07-04T18:51:23.451276+0000","last_undegraded":"2023-07-04T18:51:23.451276+0000","last_fullsized":"2023-07-04T18:51:23.451276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:02:10.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303594+0000","last_change":"2023-07-04T18:51:22.324752+0000","last_active":"2023-07-04T18:51:23.303594+0000","last_peered":"2023-07-04T18:51:23.303594+0000","last_clean":"2023-07-04T18:51:23.303594+0000","last_became_active":"2023-07-04T18:51:22.324420+0000","last_became_peered":"2023-07-04T18:51:22.324420+0000","last_unstale":"2023-07-04T18:51:23.303594+0000","last_undegraded":"2023-07-04T18:51:23.303594+0000","last_fullsized":"2023-07-04T18:51:23.303594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:06.453079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774466+0000","last_change":"2023-07-04T18:51:21.471602+0000","last_active":"2023-07-04T18:51:24.774466+0000","last_peered":"2023-07-04T18:51:24.774466+0000","last_clean":"2023-07-04T18:51:24.774466+0000","last_became_active":"2023-07-04T18:51:21.471319+0000","last_became_peered":"2023-07-04T18:51:21.471319+0000","last_unstale":"2023-07-04T18:51:24.774466+0000","last_undegraded":"2023-07-04T18:51:24.774466+0000","last_fullsized":"2023-07-04T18:51:24.774466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752233+0000","last_change":"2023-07-04T18:51:22.321087+0000","last_active":"2023-07-04T18:51:24.752233+0000","last_peered":"2023-07-04T18:51:24.752233+0000","last_clean":"2023-07-04T18:51:24.752233+0000","last_became_active":"2023-07-04T18:51:22.320871+0000","last_became_peered":"2023-07-04T18:51:22.320871+0000","last_unstale":"2023-07-04T18:51:24.752233+0000","last_undegraded":"2023-07-04T18:51:24.752233+0000","last_fullsized":"2023-07-04T18:51:24.752233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.473477+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.473173+0000","last_became_peered":"2023-07-04T18:51:21.473173+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304599+0000","last_change":"2023-07-04T18:51:22.317634+0000","last_active":"2023-07-04T18:51:23.304599+0000","last_peered":"2023-07-04T18:51:23.304599+0000","last_clean":"2023-07-04T18:51:23.304599+0000","last_became_active":"2023-07-04T18:51:22.317260+0000","last_became_peered":"2023-07-04T18:51:22.317260+0000","last_unstale":"2023-07-04T18:51:23.304599+0000","last_undegraded":"2023-07-04T18:51:23.304599+0000","last_fullsized":"2023-07-04T18:51:23.304599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303472+0000","last_change":"2023-07-04T18:51:21.463128+0000","last_active":"2023-07-04T18:51:23.303472+0000","last_peered":"2023-07-04T18:51:23.303472+0000","last_clean":"2023-07-04T18:51:23.303472+0000","last_became_active":"2023-07-04T18:51:21.462860+0000","last_became_peered":"2023-07-04T18:51:21.462860+0000","last_unstale":"2023-07-04T18:51:23.303472+0000","last_undegraded":"2023-07-04T18:51:23.303472+0000","last_fullsized":"2023-07-04T18:51:23.303472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304928+0000","last_change":"2023-07-04T18:51:22.318985+0000","last_active":"2023-07-04T18:51:23.304928+0000","last_peered":"2023-07-04T18:51:23.304928+0000","last_clean":"2023-07-04T18:51:23.304928+0000","last_became_active":"2023-07-04T18:51:22.318393+0000","last_became_peered":"2023-07-04T18:51:22.318393+0000","last_unstale":"2023-07-04T18:51:23.304928+0000","last_undegraded":"2023-07-04T18:51:23.304928+0000","last_fullsized":"2023-07-04T18:51:23.304928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304517+0000","last_change":"2023-07-04T18:51:21.466769+0000","last_active":"2023-07-04T18:51:23.304517+0000","last_peered":"2023-07-04T18:51:23.304517+0000","last_clean":"2023-07-04T18:51:23.304517+0000","last_became_active":"2023-07-04T18:51:21.466504+0000","last_became_peered":"2023-07-04T18:51:21.466504+0000","last_unstale":"2023-07-04T18:51:23.304517+0000","last_undegraded":"2023-07-04T18:51:23.304517+0000","last_fullsized":"2023-07-04T18:51:23.304517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:03.327143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304757+0000","last_change":"2023-07-04T18:51:22.317855+0000","last_active":"2023-07-04T18:51:23.304757+0000","last_peered":"2023-07-04T18:51:23.304757+0000","last_clean":"2023-07-04T18:51:23.304757+0000","last_became_active":"2023-07-04T18:51:22.317426+0000","last_became_peered":"2023-07-04T18:51:22.317426+0000","last_unstale":"2023-07-04T18:51:23.304757+0000","last_undegraded":"2023-07-04T18:51:23.304757+0000","last_fullsized":"2023-07-04T18:51:23.304757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303314+0000","last_change":"2023-07-04T18:51:21.471306+0000","last_active":"2023-07-04T18:51:23.303314+0000","last_peered":"2023-07-04T18:51:23.303314+0000","last_clean":"2023-07-04T18:51:23.303314+0000","last_became_active":"2023-07-04T18:51:21.471034+0000","last_became_peered":"2023-07-04T18:51:21.471034+0000","last_unstale":"2023-07-04T18:51:23.303314+0000","last_undegraded":"2023-07-04T18:51:23.303314+0000","last_fullsized":"2023-07-04T18:51:23.303314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417826+0000","last_change":"2023-07-04T18:51:22.315022+0000","last_active":"2023-07-04T18:51:23.417826+0000","last_peered":"2023-07-04T18:51:23.417826+0000","last_clean":"2023-07-04T18:51:23.417826+0000","last_became_active":"2023-07-04T18:51:22.314767+0000","last_became_peered":"2023-07-04T18:51:22.314767+0000","last_unstale":"2023-07-04T18:51:23.417826+0000","last_undegraded":"2023-07-04T18:51:23.417826+0000","last_fullsized":"2023-07-04T18:51:23.417826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417511+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:51:23.417511+0000","last_peered":"2023-07-04T18:51:23.417511+0000","last_clean":"2023-07-04T18:51:23.417511+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:51:23.417511+0000","last_undegraded":"2023-07-04T18:51:23.417511+0000","last_fullsized":"2023-07-04T18:51:23.417511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774644+0000","last_change":"2023-07-04T18:51:21.471879+0000","last_active":"2023-07-04T18:51:24.774644+0000","last_peered":"2023-07-04T18:51:24.774644+0000","last_clean":"2023-07-04T18:51:24.774644+0000","last_became_active":"2023-07-04T18:51:21.471549+0000","last_became_peered":"2023-07-04T18:51:21.471549+0000","last_unstale":"2023-07-04T18:51:24.774644+0000","last_undegraded":"2023-07-04T18:51:24.774644+0000","last_fullsized":"2023-07-04T18:51:24.774644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417554+0000","last_change":"2023-07-04T18:51:22.319180+0000","last_active":"2023-07-04T18:51:23.417554+0000","last_peered":"2023-07-04T18:51:23.417554+0000","last_clean":"2023-07-04T18:51:23.417554+0000","last_became_active":"2023-07-04T18:51:22.318906+0000","last_became_peered":"2023-07-04T18:51:22.318906+0000","last_unstale":"2023-07-04T18:51:23.417554+0000","last_undegraded":"2023-07-04T18:51:23.417554+0000","last_fullsized":"2023-07-04T18:51:23.417554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774410+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:51:24.774410+0000","last_peered":"2023-07-04T18:51:24.774410+0000","last_clean":"2023-07-04T18:51:24.774410+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:51:24.774410+0000","last_undegraded":"2023-07-04T18:51:24.774410+0000","last_fullsized":"2023-07-04T18:51:24.774410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:48.688640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304959+0000","last_change":"2023-07-04T18:51:21.461532+0000","last_active":"2023-07-04T18:51:23.304959+0000","last_peered":"2023-07-04T18:51:23.304959+0000","last_clean":"2023-07-04T18:51:23.304959+0000","last_became_active":"2023-07-04T18:51:21.460816+0000","last_became_peered":"2023-07-04T18:51:21.460816+0000","last_unstale":"2023-07-04T18:51:23.304959+0000","last_undegraded":"2023-07-04T18:51:23.304959+0000","last_fullsized":"2023-07-04T18:51:23.304959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417715+0000","last_change":"2023-07-04T18:51:22.312480+0000","last_active":"2023-07-04T18:51:23.417715+0000","last_peered":"2023-07-04T18:51:23.417715+0000","last_clean":"2023-07-04T18:51:23.417715+0000","last_became_active":"2023-07-04T18:51:22.311193+0000","last_became_peered":"2023-07-04T18:51:22.311193+0000","last_unstale":"2023-07-04T18:51:23.417715+0000","last_undegraded":"2023-07-04T18:51:23.417715+0000","last_fullsized":"2023-07-04T18:51:23.417715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773379+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:51:24.773379+0000","last_peered":"2023-07-04T18:51:24.773379+0000","last_clean":"2023-07-04T18:51:24.773379+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:51:24.773379+0000","last_undegraded":"2023-07-04T18:51:24.773379+0000","last_fullsized":"2023-07-04T18:51:24.773379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:15.786775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417530+0000","last_change":"2023-07-04T18:51:21.467449+0000","last_active":"2023-07-04T18:51:23.417530+0000","last_peered":"2023-07-04T18:51:23.417530+0000","last_clean":"2023-07-04T18:51:23.417530+0000","last_became_active":"2023-07-04T18:51:21.466522+0000","last_became_peered":"2023-07-04T18:51:21.466522+0000","last_unstale":"2023-07-04T18:51:23.417530+0000","last_undegraded":"2023-07-04T18:51:23.417530+0000","last_fullsized":"2023-07-04T18:51:23.417530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417692+0000","last_change":"2023-07-04T18:51:22.324258+0000","last_active":"2023-07-04T18:51:23.417692+0000","last_peered":"2023-07-04T18:51:23.417692+0000","last_clean":"2023-07-04T18:51:23.417692+0000","last_became_active":"2023-07-04T18:51:22.323966+0000","last_became_peered":"2023-07-04T18:51:22.323966+0000","last_unstale":"2023-07-04T18:51:23.417692+0000","last_undegraded":"2023-07-04T18:51:23.417692+0000","last_fullsized":"2023-07-04T18:51:23.417692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774412+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:51:24.774412+0000","last_peered":"2023-07-04T18:51:24.774412+0000","last_clean":"2023-07-04T18:51:24.774412+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:51:24.774412+0000","last_undegraded":"2023-07-04T18:51:24.774412+0000","last_fullsized":"2023-07-04T18:51:24.774412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450698+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:51:23.450698+0000","last_peered":"2023-07-04T18:51:23.450698+0000","last_clean":"2023-07-04T18:51:23.450698+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:51:23.450698+0000","last_undegraded":"2023-07-04T18:51:23.450698+0000","last_fullsized":"2023-07-04T18:51:23.450698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:37.197069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303756+0000","last_change":"2023-07-04T18:51:21.470626+0000","last_active":"2023-07-04T18:51:23.303756+0000","last_peered":"2023-07-04T18:51:23.303756+0000","last_clean":"2023-07-04T18:51:23.303756+0000","last_became_active":"2023-07-04T18:51:21.470351+0000","last_became_peered":"2023-07-04T18:51:21.470351+0000","last_unstale":"2023-07-04T18:51:23.303756+0000","last_undegraded":"2023-07-04T18:51:23.303756+0000","last_fullsized":"2023-07-04T18:51:23.303756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303308+0000","last_change":"2023-07-04T18:51:22.321449+0000","last_active":"2023-07-04T18:51:23.303308+0000","last_peered":"2023-07-04T18:51:23.303308+0000","last_clean":"2023-07-04T18:51:23.303308+0000","last_became_active":"2023-07-04T18:51:22.321158+0000","last_became_peered":"2023-07-04T18:51:22.321158+0000","last_unstale":"2023-07-04T18:51:23.303308+0000","last_undegraded":"2023-07-04T18:51:23.303308+0000","last_fullsized":"2023-07-04T18:51:23.303308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450897+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:51:23.450897+0000","last_peered":"2023-07-04T18:51:23.450897+0000","last_clean":"2023-07-04T18:51:23.450897+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:51:23.450897+0000","last_undegraded":"2023-07-04T18:51:23.450897+0000","last_fullsized":"2023-07-04T18:51:23.450897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:48.113610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752148+0000","last_change":"2023-07-04T18:51:21.460314+0000","last_active":"2023-07-04T18:51:24.752148+0000","last_peered":"2023-07-04T18:51:24.752148+0000","last_clean":"2023-07-04T18:51:24.752148+0000","last_became_active":"2023-07-04T18:51:21.460022+0000","last_became_peered":"2023-07-04T18:51:21.460022+0000","last_unstale":"2023-07-04T18:51:24.752148+0000","last_undegraded":"2023-07-04T18:51:24.752148+0000","last_fullsized":"2023-07-04T18:51:24.752148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:50.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752641+0000","last_change":"2023-07-04T18:51:22.321571+0000","last_active":"2023-07-04T18:51:24.752641+0000","last_peered":"2023-07-04T18:51:24.752641+0000","last_clean":"2023-07-04T18:51:24.752641+0000","last_became_active":"2023-07-04T18:51:22.321326+0000","last_became_peered":"2023-07-04T18:51:22.321326+0000","last_unstale":"2023-07-04T18:51:24.752641+0000","last_undegraded":"2023-07-04T18:51:24.752641+0000","last_fullsized":"2023-07-04T18:51:24.752641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451151+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:51:23.451151+0000","last_peered":"2023-07-04T18:51:23.451151+0000","last_clean":"2023-07-04T18:51:23.451151+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:51:23.451151+0000","last_undegraded":"2023-07-04T18:51:23.451151+0000","last_fullsized":"2023-07-04T18:51:23.451151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:14:08.639700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417718+0000","last_change":"2023-07-04T18:51:21.466845+0000","last_active":"2023-07-04T18:51:23.417718+0000","last_peered":"2023-07-04T18:51:23.417718+0000","last_clean":"2023-07-04T18:51:23.417718+0000","last_became_active":"2023-07-04T18:51:21.466546+0000","last_became_peered":"2023-07-04T18:51:21.466546+0000","last_unstale":"2023-07-04T18:51:23.417718+0000","last_undegraded":"2023-07-04T18:51:23.417718+0000","last_fullsized":"2023-07-04T18:51:23.417718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752539+0000","last_change":"2023-07-04T18:51:22.318511+0000","last_active":"2023-07-04T18:51:24.752539+0000","last_peered":"2023-07-04T18:51:24.752539+0000","last_clean":"2023-07-04T18:51:24.752539+0000","last_became_active":"2023-07-04T18:51:22.318227+0000","last_became_peered":"2023-07-04T18:51:22.318227+0000","last_unstale":"2023-07-04T18:51:24.752539+0000","last_undegraded":"2023-07-04T18:51:24.752539+0000","last_fullsized":"2023-07-04T18:51:24.752539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304031+0000","last_change":"2023-07-04T18:51:21.474661+0000","last_active":"2023-07-04T18:51:23.304031+0000","last_peered":"2023-07-04T18:51:23.304031+0000","last_clean":"2023-07-04T18:51:23.304031+0000","last_became_active":"2023-07-04T18:51:21.474400+0000","last_became_peered":"2023-07-04T18:51:21.474400+0000","last_unstale":"2023-07-04T18:51:23.304031+0000","last_undegraded":"2023-07-04T18:51:23.304031+0000","last_fullsized":"2023-07-04T18:51:23.304031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417697+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:51:23.417697+0000","last_peered":"2023-07-04T18:51:23.417697+0000","last_clean":"2023-07-04T18:51:23.417697+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:51:23.417697+0000","last_undegraded":"2023-07-04T18:51:23.417697+0000","last_fullsized":"2023-07-04T18:51:23.417697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:02.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303243+0000","last_change":"2023-07-04T18:51:22.313754+0000","last_active":"2023-07-04T18:51:23.303243+0000","last_peered":"2023-07-04T18:51:23.303243+0000","last_clean":"2023-07-04T18:51:23.303243+0000","last_became_active":"2023-07-04T18:51:22.313509+0000","last_became_peered":"2023-07-04T18:51:22.313509+0000","last_unstale":"2023-07-04T18:51:23.303243+0000","last_undegraded":"2023-07-04T18:51:23.303243+0000","last_fullsized":"2023-07-04T18:51:23.303243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303419+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:51:23.303419+0000","last_peered":"2023-07-04T18:51:23.303419+0000","last_clean":"2023-07-04T18:51:23.303419+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:51:23.303419+0000","last_undegraded":"2023-07-04T18:51:23.303419+0000","last_fullsized":"2023-07-04T18:51:23.303419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752666+0000","last_change":"2023-07-04T18:51:21.464739+0000","last_active":"2023-07-04T18:51:24.752666+0000","last_peered":"2023-07-04T18:51:24.752666+0000","last_clean":"2023-07-04T18:51:24.752666+0000","last_became_active":"2023-07-04T18:51:21.464445+0000","last_became_peered":"2023-07-04T18:51:21.464445+0000","last_unstale":"2023-07-04T18:51:24.752666+0000","last_undegraded":"2023-07-04T18:51:24.752666+0000","last_fullsized":"2023-07-04T18:51:24.752666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304372+0000","last_change":"2023-07-04T18:51:22.326679+0000","last_active":"2023-07-04T18:51:23.304372+0000","last_peered":"2023-07-04T18:51:23.304372+0000","last_clean":"2023-07-04T18:51:23.304372+0000","last_became_active":"2023-07-04T18:51:22.326348+0000","last_became_peered":"2023-07-04T18:51:22.326348+0000","last_unstale":"2023-07-04T18:51:23.304372+0000","last_undegraded":"2023-07-04T18:51:23.304372+0000","last_fullsized":"2023-07-04T18:51:23.304372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752305+0000","last_change":"2023-07-04T18:51:21.466007+0000","last_active":"2023-07-04T18:51:24.752305+0000","last_peered":"2023-07-04T18:51:24.752305+0000","last_clean":"2023-07-04T18:51:24.752305+0000","last_became_active":"2023-07-04T18:51:21.465754+0000","last_became_peered":"2023-07-04T18:51:21.465754+0000","last_unstale":"2023-07-04T18:51:24.752305+0000","last_undegraded":"2023-07-04T18:51:24.752305+0000","last_fullsized":"2023-07-04T18:51:24.752305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773596+0000","last_change":"2023-07-04T18:51:22.309513+0000","last_active":"2023-07-04T18:51:24.773596+0000","last_peered":"2023-07-04T18:51:24.773596+0000","last_clean":"2023-07-04T18:51:24.773596+0000","last_became_active":"2023-07-04T18:51:22.309252+0000","last_became_peered":"2023-07-04T18:51:22.309252+0000","last_unstale":"2023-07-04T18:51:24.773596+0000","last_undegraded":"2023-07-04T18:51:24.773596+0000","last_fullsized":"2023-07-04T18:51:24.773596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774594+0000","last_change":"2023-07-04T18:51:21.463261+0000","last_active":"2023-07-04T18:51:24.774594+0000","last_peered":"2023-07-04T18:51:24.774594+0000","last_clean":"2023-07-04T18:51:24.774594+0000","last_became_active":"2023-07-04T18:51:21.462985+0000","last_became_peered":"2023-07-04T18:51:21.462985+0000","last_unstale":"2023-07-04T18:51:24.774594+0000","last_undegraded":"2023-07-04T18:51:24.774594+0000","last_fullsized":"2023-07-04T18:51:24.774594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:47.501589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304859+0000","last_change":"2023-07-04T18:51:22.318260+0000","last_active":"2023-07-04T18:51:23.304859+0000","last_peered":"2023-07-04T18:51:23.304859+0000","last_clean":"2023-07-04T18:51:23.304859+0000","last_became_active":"2023-07-04T18:51:22.317037+0000","last_became_peered":"2023-07-04T18:51:22.317037+0000","last_unstale":"2023-07-04T18:51:23.304859+0000","last_undegraded":"2023-07-04T18:51:23.304859+0000","last_fullsized":"2023-07-04T18:51:23.304859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774762+0000","last_change":"2023-07-04T18:51:21.463234+0000","last_active":"2023-07-04T18:51:24.774762+0000","last_peered":"2023-07-04T18:51:24.774762+0000","last_clean":"2023-07-04T18:51:24.774762+0000","last_became_active":"2023-07-04T18:51:21.462954+0000","last_became_peered":"2023-07-04T18:51:21.462954+0000","last_unstale":"2023-07-04T18:51:24.774762+0000","last_undegraded":"2023-07-04T18:51:24.774762+0000","last_fullsized":"2023-07-04T18:51:24.774762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:52.205620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304612+0000","last_change":"2023-07-04T18:51:22.317628+0000","last_active":"2023-07-04T18:51:23.304612+0000","last_peered":"2023-07-04T18:51:23.304612+0000","last_clean":"2023-07-04T18:51:23.304612+0000","last_became_active":"2023-07-04T18:51:22.317043+0000","last_became_peered":"2023-07-04T18:51:22.317043+0000","last_unstale":"2023-07-04T18:51:23.304612+0000","last_undegraded":"2023-07-04T18:51:23.304612+0000","last_fullsized":"2023-07-04T18:51:23.304612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.470557+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.470247+0000","last_became_peered":"2023-07-04T18:51:21.470247+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:13.171949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303469+0000","last_change":"2023-07-04T18:51:22.313146+0000","last_active":"2023-07-04T18:51:23.303469+0000","last_peered":"2023-07-04T18:51:23.303469+0000","last_clean":"2023-07-04T18:51:23.303469+0000","last_became_active":"2023-07-04T18:51:22.312877+0000","last_became_peered":"2023-07-04T18:51:22.312877+0000","last_unstale":"2023-07-04T18:51:23.303469+0000","last_undegraded":"2023-07-04T18:51:23.303469+0000","last_fullsized":"2023-07-04T18:51:23.303469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752619+0000","last_change":"2023-07-04T18:51:21.459772+0000","last_active":"2023-07-04T18:51:24.752619+0000","last_peered":"2023-07-04T18:51:24.752619+0000","last_clean":"2023-07-04T18:51:24.752619+0000","last_became_active":"2023-07-04T18:51:21.459459+0000","last_became_peered":"2023-07-04T18:51:21.459459+0000","last_unstale":"2023-07-04T18:51:24.752619+0000","last_undegraded":"2023-07-04T18:51:24.752619+0000","last_fullsized":"2023-07-04T18:51:24.752619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:47.501589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303922+0000","last_change":"2023-07-04T18:51:22.318229+0000","last_active":"2023-07-04T18:51:23.303922+0000","last_peered":"2023-07-04T18:51:23.303922+0000","last_clean":"2023-07-04T18:51:23.303922+0000","last_became_active":"2023-07-04T18:51:22.317388+0000","last_became_peered":"2023-07-04T18:51:22.317388+0000","last_unstale":"2023-07-04T18:51:23.303922+0000","last_undegraded":"2023-07-04T18:51:23.303922+0000","last_fullsized":"2023-07-04T18:51:23.303922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:55.968814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303155+0000","last_change":"2023-07-04T18:51:22.314933+0000","last_active":"2023-07-04T18:51:23.303155+0000","last_peered":"2023-07-04T18:51:23.303155+0000","last_clean":"2023-07-04T18:51:23.303155+0000","last_became_active":"2023-07-04T18:51:22.314648+0000","last_became_peered":"2023-07-04T18:51:22.314648+0000","last_unstale":"2023-07-04T18:51:23.303155+0000","last_undegraded":"2023-07-04T18:51:23.303155+0000","last_fullsized":"2023-07-04T18:51:23.303155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303365+0000","last_change":"2023-07-04T18:51:22.318422+0000","last_active":"2023-07-04T18:51:23.303365+0000","last_peered":"2023-07-04T18:51:23.303365+0000","last_clean":"2023-07-04T18:51:23.303365+0000","last_became_active":"2023-07-04T18:51:22.318161+0000","last_became_peered":"2023-07-04T18:51:22.318161+0000","last_unstale":"2023-07-04T18:51:23.303365+0000","last_undegraded":"2023-07-04T18:51:23.303365+0000","last_fullsized":"2023-07-04T18:51:23.303365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304314+0000","last_change":"2023-07-04T18:51:22.320144+0000","last_active":"2023-07-04T18:51:23.304314+0000","last_peered":"2023-07-04T18:51:23.304314+0000","last_clean":"2023-07-04T18:51:23.304314+0000","last_became_active":"2023-07-04T18:51:22.317892+0000","last_became_peered":"2023-07-04T18:51:22.317892+0000","last_unstale":"2023-07-04T18:51:23.304314+0000","last_undegraded":"2023-07-04T18:51:23.304314+0000","last_fullsized":"2023-07-04T18:51:23.304314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303934+0000","last_change":"2023-07-04T18:51:22.317154+0000","last_active":"2023-07-04T18:51:23.303934+0000","last_peered":"2023-07-04T18:51:23.303934+0000","last_clean":"2023-07-04T18:51:23.303934+0000","last_became_active":"2023-07-04T18:51:22.316916+0000","last_became_peered":"2023-07-04T18:51:22.316916+0000","last_unstale":"2023-07-04T18:51:23.303934+0000","last_undegraded":"2023-07-04T18:51:23.303934+0000","last_fullsized":"2023-07-04T18:51:23.303934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417430+0000","last_change":"2023-07-04T18:51:22.312307+0000","last_active":"2023-07-04T18:51:23.417430+0000","last_peered":"2023-07-04T18:51:23.417430+0000","last_clean":"2023-07-04T18:51:23.417430+0000","last_became_active":"2023-07-04T18:51:22.310832+0000","last_became_peered":"2023-07-04T18:51:22.310832+0000","last_unstale":"2023-07-04T18:51:23.417430+0000","last_undegraded":"2023-07-04T18:51:23.417430+0000","last_fullsized":"2023-07-04T18:51:23.417430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":864256,"data_stored":681747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7719999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.742}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":262144,"data_stored":91219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":249856,"data_stored":90251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:51:31.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:51:31.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.381+0000 7fab491bb700 1 Processor -- start 2023-07-04T18:51:31.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.381+0000 7fab491bb700 1 -- start start 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 0x7fab440cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab4415ca10 0x7fab44070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab44071210 con 0x7fab4415ca10 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab440717f0 con 0x7fab4415c640 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab491bb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab440cd290 con 0x7fab44070db0 2023-07-04T18:51:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab4415ca10 0x7fab44070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fab4415ca10 0x7fab44070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44364/0 (socket says 172.21.15.77:44364) 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 -- 172.21.15.77:0/111500292 learned_addr learned my addr 172.21.15.77:0/111500292 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab42d9d700 1 --2- 172.21.15.77:0/111500292 >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 0x7fab440cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4359e700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 -- 172.21.15.77:0/111500292 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 msgr2=0x7fab440cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 -- 172.21.15.77:0/111500292 >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 msgr2=0x7fab440cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 --2- 172.21.15.77:0/111500292 >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 0x7fab440cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab4259c700 1 -- 172.21.15.77:0/111500292 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440ccfb0 con 0x7fab4415ca10 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.382+0000 7fab42d9d700 1 --2- 172.21.15.77:0/111500292 >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 0x7fab440cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab4259c700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415ca10 0x7fab44070870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fab44069540 tx=0x7fab34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e0a311ae6217df server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab41d9b700 1 -- 172.21.15.77:0/111500292 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400b870 con 0x7fab4415ca10 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab41d9b700 1 -- 172.21.15.77:0/111500292 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3400b9d0 con 0x7fab4415ca10 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab41d9b700 1 -- 172.21.15.77:0/111500292 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400bd60 con 0x7fab4415ca10 2023-07-04T18:51:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 -- 172.21.15.77:0/111500292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415ca10 msgr2=0x7fab44070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415ca10 0x7fab44070870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fab44069540 tx=0x7fab34009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 -- 172.21.15.77:0/111500292 shutdown_connections 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 --2- 172.21.15.77:0/111500292 >> v2:172.21.15.82:3300/0 conn(0x7fab4415c640 0x7fab440cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 --2- 172.21.15.77:0/111500292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415ca10 0x7fab44070870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 -- 172.21.15.77:0/111500292 >> 172.21.15.77:0/111500292 conn(0x7fab4406bba0 msgr2=0x7fab440b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 -- 172.21.15.77:0/111500292 shutdown_connections 2023-07-04T18:51:31.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.383+0000 7fab491bb700 1 -- 172.21.15.77:0/111500292 wait complete. 2023-07-04T18:51:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.385+0000 7fab491bb700 1 Processor -- start 2023-07-04T18:51:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.385+0000 7fab491bb700 1 -- start start 2023-07-04T18:51:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4415ca10 0x7fab440cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440c8cf0 con 0x7fab4415c640 2023-07-04T18:51:31.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440b4b80 con 0x7fab4415ca10 2023-07-04T18:51:31.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab491bb700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab440b4e60 con 0x7fab44070db0 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab4259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab4259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44370/0 (socket says 172.21.15.77:44370) 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab4259c700 1 -- 172.21.15.77:0/1706571595 learned_addr learned my addr 172.21.15.77:0/1706571595 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab4359e700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4415ca10 0x7fab440cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.386+0000 7fab42d9d700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 -- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 msgr2=0x7fab440b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 -- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4415ca10 msgr2=0x7fab440cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4415ca10 0x7fab440cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 -- 172.21.15.77:0/1706571595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34009550 con 0x7fab4415c640 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab4259c700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab34009c10 tx=0x7fab34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab340036a0 con 0x7fab4415c640 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34003800 con 0x7fab4415c640 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab440b5140 con 0x7fab4415c640 2023-07-04T18:51:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34003bb0 con 0x7fab4415c640 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.387+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab440b5700 con 0x7fab4415c640 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.388+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4404e210 con 0x7fab4415c640 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.389+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fab34003d10 con 0x7fab4415c640 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.389+0000 7fab2b7fe700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 0x7fab3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.389+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fab3408abf0 con 0x7fab4415c640 2023-07-04T18:51:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.389+0000 7fab42d9d700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 0x7fab3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.389+0000 7fab42d9d700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 0x7fab3006d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fab38009870 tx=0x7fab38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.393+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fab34055f30 con 0x7fab4415c640 2023-07-04T18:51:31.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.577+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab440ce920 con 0x7fab3006ace0 2023-07-04T18:51:31.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.585+0000 7fab2b7fe700 1 -- 172.21.15.77:0/1706571595 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286620 (secure 0 0 0) 0x7fab3800c050 con 0x7fab3006ace0 2023-07-04T18:51:31.590 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:31.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 msgr2=0x7fab3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 0x7fab3006d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fab38009870 tx=0x7fab38008040 comp rx=0 tx=0).stop 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 msgr2=0x7fab440cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab34009c10 tx=0x7fab34004750 comp rx=0 tx=0).stop 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 shutdown_connections 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fab3006ace0 0x7fab3006d190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab4415ca10 0x7fab440cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fab44070db0 0x7fab440b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 --2- 172.21.15.77:0/1706571595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab4415c640 0x7fab440cf020 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 >> 172.21.15.77:0/1706571595 conn(0x7fab4406bba0 msgr2=0x7fab440b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 shutdown_connections 2023-07-04T18:51:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.588+0000 7fab491bb700 1 -- 172.21.15.77:0/1706571595 wait complete. 2023-07-04T18:51:31.594 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:51:31.602 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-07-04T18:51:31.178587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218728,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755992,"statfs":{"total":773094113280,"available":772870135808,"internally_reserved":0,"allocated":3186688,"data_stored":1902252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.875834"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773578+0000","last_change":"2023-07-04T18:51:22.314047+0000","last_active":"2023-07-04T18:51:24.773578+0000","last_peered":"2023-07-04T18:51:24.773578+0000","last_clean":"2023-07-04T18:51:24.773578+0000","last_became_active":"2023-07-04T18:51:22.313772+0000","last_became_peered":"2023-07-04T18:51:22.313772+0000","last_unstale":"2023-07-04T18:51:24.773578+0000","last_undegraded":"2023-07-04T18:51:24.773578+0000","last_fullsized":"2023-07-04T18:51:24.773578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:09.444946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774630+0000","last_change":"2023-07-04T18:51:22.324407+0000","last_active":"2023-07-04T18:51:24.774630+0000","last_peered":"2023-07-04T18:51:24.774630+0000","last_clean":"2023-07-04T18:51:24.774630+0000","last_became_active":"2023-07-04T18:51:22.323862+0000","last_became_peered":"2023-07-04T18:51:22.323862+0000","last_unstale":"2023-07-04T18:51:24.774630+0000","last_undegraded":"2023-07-04T18:51:24.774630+0000","last_fullsized":"2023-07-04T18:51:24.774630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304530+0000","last_change":"2023-07-04T18:51:22.319081+0000","last_active":"2023-07-04T18:51:23.304530+0000","last_peered":"2023-07-04T18:51:23.304530+0000","last_clean":"2023-07-04T18:51:23.304530+0000","last_became_active":"2023-07-04T18:51:22.318806+0000","last_became_peered":"2023-07-04T18:51:22.318806+0000","last_unstale":"2023-07-04T18:51:23.304530+0000","last_undegraded":"2023-07-04T18:51:23.304530+0000","last_fullsized":"2023-07-04T18:51:23.304530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303285+0000","last_change":"2023-07-04T18:51:22.319728+0000","last_active":"2023-07-04T18:51:23.303285+0000","last_peered":"2023-07-04T18:51:23.303285+0000","last_clean":"2023-07-04T18:51:23.303285+0000","last_became_active":"2023-07-04T18:51:22.319417+0000","last_became_peered":"2023-07-04T18:51:22.319417+0000","last_unstale":"2023-07-04T18:51:23.303285+0000","last_undegraded":"2023-07-04T18:51:23.303285+0000","last_fullsized":"2023-07-04T18:51:23.303285+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752506+0000","last_change":"2023-07-04T18:51:22.317945+0000","last_active":"2023-07-04T18:51:24.752506+0000","last_peered":"2023-07-04T18:51:24.752506+0000","last_clean":"2023-07-04T18:51:24.752506+0000","last_became_active":"2023-07-04T18:51:22.317635+0000","last_became_peered":"2023-07-04T18:51:22.317635+0000","last_unstale":"2023-07-04T18:51:24.752506+0000","last_undegraded":"2023-07-04T18:51:24.752506+0000","last_fullsized":"2023-07-04T18:51:24.752506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304259+0000","last_change":"2023-07-04T18:51:22.320918+0000","last_active":"2023-07-04T18:51:23.304259+0000","last_peered":"2023-07-04T18:51:23.304259+0000","last_clean":"2023-07-04T18:51:23.304259+0000","last_became_active":"2023-07-04T18:51:22.320502+0000","last_became_peered":"2023-07-04T18:51:22.320502+0000","last_unstale":"2023-07-04T18:51:23.304259+0000","last_undegraded":"2023-07-04T18:51:23.304259+0000","last_fullsized":"2023-07-04T18:51:23.304259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450875+0000","last_change":"2023-07-04T18:51:22.323607+0000","last_active":"2023-07-04T18:51:23.450875+0000","last_peered":"2023-07-04T18:51:23.450875+0000","last_clean":"2023-07-04T18:51:23.450875+0000","last_became_active":"2023-07-04T18:51:22.322940+0000","last_became_peered":"2023-07-04T18:51:22.322940+0000","last_unstale":"2023-07-04T18:51:23.450875+0000","last_undegraded":"2023-07-04T18:51:23.450875+0000","last_fullsized":"2023-07-04T18:51:23.450875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774287+0000","last_change":"2023-07-04T18:51:22.321665+0000","last_active":"2023-07-04T18:51:24.774287+0000","last_peered":"2023-07-04T18:51:24.774287+0000","last_clean":"2023-07-04T18:51:24.774287+0000","last_became_active":"2023-07-04T18:51:22.321426+0000","last_became_peered":"2023-07-04T18:51:22.321426+0000","last_unstale":"2023-07-04T18:51:24.774287+0000","last_undegraded":"2023-07-04T18:51:24.774287+0000","last_fullsized":"2023-07-04T18:51:24.774287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752371+0000","last_change":"2023-07-04T18:51:22.318535+0000","last_active":"2023-07-04T18:51:24.752371+0000","last_peered":"2023-07-04T18:51:24.752371+0000","last_clean":"2023-07-04T18:51:24.752371+0000","last_became_active":"2023-07-04T18:51:22.318160+0000","last_became_peered":"2023-07-04T18:51:22.318160+0000","last_unstale":"2023-07-04T18:51:24.752371+0000","last_undegraded":"2023-07-04T18:51:24.752371+0000","last_fullsized":"2023-07-04T18:51:24.752371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774802+0000","last_change":"2023-07-04T18:51:22.319854+0000","last_active":"2023-07-04T18:51:24.774802+0000","last_peered":"2023-07-04T18:51:24.774802+0000","last_clean":"2023-07-04T18:51:24.774802+0000","last_became_active":"2023-07-04T18:51:22.319581+0000","last_became_peered":"2023-07-04T18:51:22.319581+0000","last_unstale":"2023-07-04T18:51:24.774802+0000","last_undegraded":"2023-07-04T18:51:24.774802+0000","last_fullsized":"2023-07-04T18:51:24.774802+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:28.243222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304943+0000","last_change":"2023-07-04T18:51:22.322222+0000","last_active":"2023-07-04T18:51:23.304943+0000","last_peered":"2023-07-04T18:51:23.304943+0000","last_clean":"2023-07-04T18:51:23.304943+0000","last_became_active":"2023-07-04T18:51:22.321944+0000","last_became_peered":"2023-07-04T18:51:22.321944+0000","last_unstale":"2023-07-04T18:51:23.304943+0000","last_undegraded":"2023-07-04T18:51:23.304943+0000","last_fullsized":"2023-07-04T18:51:23.304943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:53.251606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773693+0000","last_change":"2023-07-04T18:51:22.318773+0000","last_active":"2023-07-04T18:51:24.773693+0000","last_peered":"2023-07-04T18:51:24.773693+0000","last_clean":"2023-07-04T18:51:24.773693+0000","last_became_active":"2023-07-04T18:51:22.318495+0000","last_became_peered":"2023-07-04T18:51:22.318495+0000","last_unstale":"2023-07-04T18:51:24.773693+0000","last_undegraded":"2023-07-04T18:51:24.773693+0000","last_fullsized":"2023-07-04T18:51:24.773693+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773431+0000","last_change":"2023-07-04T18:51:22.313313+0000","last_active":"2023-07-04T18:51:24.773431+0000","last_peered":"2023-07-04T18:51:24.773431+0000","last_clean":"2023-07-04T18:51:24.773431+0000","last_became_active":"2023-07-04T18:51:22.313049+0000","last_became_peered":"2023-07-04T18:51:22.313049+0000","last_unstale":"2023-07-04T18:51:24.773431+0000","last_undegraded":"2023-07-04T18:51:24.773431+0000","last_fullsized":"2023-07-04T18:51:24.773431+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:58:18.636942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774464+0000","last_change":"2023-07-04T18:51:22.322812+0000","last_active":"2023-07-04T18:51:24.774464+0000","last_peered":"2023-07-04T18:51:24.774464+0000","last_clean":"2023-07-04T18:51:24.774464+0000","last_became_active":"2023-07-04T18:51:22.318629+0000","last_became_peered":"2023-07-04T18:51:22.318629+0000","last_unstale":"2023-07-04T18:51:24.774464+0000","last_undegraded":"2023-07-04T18:51:24.774464+0000","last_fullsized":"2023-07-04T18:51:24.774464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450996+0000","last_change":"2023-07-04T18:51:22.323740+0000","last_active":"2023-07-04T18:51:23.450996+0000","last_peered":"2023-07-04T18:51:23.450996+0000","last_clean":"2023-07-04T18:51:23.450996+0000","last_became_active":"2023-07-04T18:51:22.323321+0000","last_became_peered":"2023-07-04T18:51:22.323321+0000","last_unstale":"2023-07-04T18:51:23.450996+0000","last_undegraded":"2023-07-04T18:51:23.450996+0000","last_fullsized":"2023-07-04T18:51:23.450996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:33:48.187398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304015+0000","last_change":"2023-07-04T18:51:22.318780+0000","last_active":"2023-07-04T18:51:23.304015+0000","last_peered":"2023-07-04T18:51:23.304015+0000","last_clean":"2023-07-04T18:51:23.304015+0000","last_became_active":"2023-07-04T18:51:22.318460+0000","last_became_peered":"2023-07-04T18:51:22.318460+0000","last_unstale":"2023-07-04T18:51:23.304015+0000","last_undegraded":"2023-07-04T18:51:23.304015+0000","last_fullsized":"2023-07-04T18:51:23.304015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752520+0000","last_change":"2023-07-04T18:51:22.318170+0000","last_active":"2023-07-04T18:51:24.752520+0000","last_peered":"2023-07-04T18:51:24.752520+0000","last_clean":"2023-07-04T18:51:24.752520+0000","last_became_active":"2023-07-04T18:51:22.317879+0000","last_became_peered":"2023-07-04T18:51:22.317879+0000","last_unstale":"2023-07-04T18:51:24.752520+0000","last_undegraded":"2023-07-04T18:51:24.752520+0000","last_fullsized":"2023-07-04T18:51:24.752520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304493+0000","last_change":"2023-07-04T18:51:22.318559+0000","last_active":"2023-07-04T18:51:23.304493+0000","last_peered":"2023-07-04T18:51:23.304493+0000","last_clean":"2023-07-04T18:51:23.304493+0000","last_became_active":"2023-07-04T18:51:22.317739+0000","last_became_peered":"2023-07-04T18:51:22.317739+0000","last_unstale":"2023-07-04T18:51:23.304493+0000","last_undegraded":"2023-07-04T18:51:23.304493+0000","last_fullsized":"2023-07-04T18:51:23.304493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773617+0000","last_change":"2023-07-04T18:51:22.310959+0000","last_active":"2023-07-04T18:51:24.773617+0000","last_peered":"2023-07-04T18:51:24.773617+0000","last_clean":"2023-07-04T18:51:24.773617+0000","last_became_active":"2023-07-04T18:51:22.310668+0000","last_became_peered":"2023-07-04T18:51:22.310668+0000","last_unstale":"2023-07-04T18:51:24.773617+0000","last_undegraded":"2023-07-04T18:51:24.773617+0000","last_fullsized":"2023-07-04T18:51:24.773617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:53:07.864187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304398+0000","last_change":"2023-07-04T18:51:22.318672+0000","last_active":"2023-07-04T18:51:23.304398+0000","last_peered":"2023-07-04T18:51:23.304398+0000","last_clean":"2023-07-04T18:51:23.304398+0000","last_became_active":"2023-07-04T18:51:22.317856+0000","last_became_peered":"2023-07-04T18:51:22.317856+0000","last_unstale":"2023-07-04T18:51:23.304398+0000","last_undegraded":"2023-07-04T18:51:23.304398+0000","last_fullsized":"2023-07-04T18:51:23.304398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451144+0000","last_change":"2023-07-04T18:51:22.321788+0000","last_active":"2023-07-04T18:51:23.451144+0000","last_peered":"2023-07-04T18:51:23.451144+0000","last_clean":"2023-07-04T18:51:23.451144+0000","last_became_active":"2023-07-04T18:51:22.321394+0000","last_became_peered":"2023-07-04T18:51:22.321394+0000","last_unstale":"2023-07-04T18:51:23.451144+0000","last_undegraded":"2023-07-04T18:51:23.451144+0000","last_fullsized":"2023-07-04T18:51:23.451144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:00:25.437637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752367+0000","last_change":"2023-07-04T18:51:22.321742+0000","last_active":"2023-07-04T18:51:24.752367+0000","last_peered":"2023-07-04T18:51:24.752367+0000","last_clean":"2023-07-04T18:51:24.752367+0000","last_became_active":"2023-07-04T18:51:22.321526+0000","last_became_peered":"2023-07-04T18:51:22.321526+0000","last_unstale":"2023-07-04T18:51:24.752367+0000","last_undegraded":"2023-07-04T18:51:24.752367+0000","last_fullsized":"2023-07-04T18:51:24.752367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303613+0000","last_change":"2023-07-04T18:51:22.321095+0000","last_active":"2023-07-04T18:51:23.303613+0000","last_peered":"2023-07-04T18:51:23.303613+0000","last_clean":"2023-07-04T18:51:23.303613+0000","last_became_active":"2023-07-04T18:51:22.320848+0000","last_became_peered":"2023-07-04T18:51:22.320848+0000","last_unstale":"2023-07-04T18:51:23.303613+0000","last_undegraded":"2023-07-04T18:51:23.303613+0000","last_fullsized":"2023-07-04T18:51:23.303613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450972+0000","last_change":"2023-07-04T18:51:22.322035+0000","last_active":"2023-07-04T18:51:23.450972+0000","last_peered":"2023-07-04T18:51:23.450972+0000","last_clean":"2023-07-04T18:51:23.450972+0000","last_became_active":"2023-07-04T18:51:22.321725+0000","last_became_peered":"2023-07-04T18:51:22.321725+0000","last_unstale":"2023-07-04T18:51:23.450972+0000","last_undegraded":"2023-07-04T18:51:23.450972+0000","last_fullsized":"2023-07-04T18:51:23.450972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417751+0000","last_change":"2023-07-04T18:51:22.311616+0000","last_active":"2023-07-04T18:51:23.417751+0000","last_peered":"2023-07-04T18:51:23.417751+0000","last_clean":"2023-07-04T18:51:23.417751+0000","last_became_active":"2023-07-04T18:51:22.311344+0000","last_became_peered":"2023-07-04T18:51:22.311344+0000","last_unstale":"2023-07-04T18:51:23.417751+0000","last_undegraded":"2023-07-04T18:51:23.417751+0000","last_fullsized":"2023-07-04T18:51:23.417751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417648+0000","last_change":"2023-07-04T18:51:22.311562+0000","last_active":"2023-07-04T18:51:23.417648+0000","last_peered":"2023-07-04T18:51:23.417648+0000","last_clean":"2023-07-04T18:51:23.417648+0000","last_became_active":"2023-07-04T18:51:22.311234+0000","last_became_peered":"2023-07-04T18:51:22.311234+0000","last_unstale":"2023-07-04T18:51:23.417648+0000","last_undegraded":"2023-07-04T18:51:23.417648+0000","last_fullsized":"2023-07-04T18:51:23.417648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774505+0000","last_change":"2023-07-04T18:51:22.325120+0000","last_active":"2023-07-04T18:51:24.774505+0000","last_peered":"2023-07-04T18:51:24.774505+0000","last_clean":"2023-07-04T18:51:24.774505+0000","last_became_active":"2023-07-04T18:51:22.324834+0000","last_became_peered":"2023-07-04T18:51:22.324834+0000","last_unstale":"2023-07-04T18:51:24.774505+0000","last_undegraded":"2023-07-04T18:51:24.774505+0000","last_fullsized":"2023-07-04T18:51:24.774505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304437+0000","last_change":"2023-07-04T18:51:21.460823+0000","last_active":"2023-07-04T18:51:23.304437+0000","last_peered":"2023-07-04T18:51:23.304437+0000","last_clean":"2023-07-04T18:51:23.304437+0000","last_became_active":"2023-07-04T18:51:21.460492+0000","last_became_peered":"2023-07-04T18:51:21.460492+0000","last_unstale":"2023-07-04T18:51:23.304437+0000","last_undegraded":"2023-07-04T18:51:23.304437+0000","last_fullsized":"2023-07-04T18:51:23.304437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303755+0000","last_change":"2023-07-04T18:51:22.313093+0000","last_active":"2023-07-04T18:51:23.303755+0000","last_peered":"2023-07-04T18:51:23.303755+0000","last_clean":"2023-07-04T18:51:23.303755+0000","last_became_active":"2023-07-04T18:51:22.312832+0000","last_became_peered":"2023-07-04T18:51:22.312832+0000","last_unstale":"2023-07-04T18:51:23.303755+0000","last_undegraded":"2023-07-04T18:51:23.303755+0000","last_fullsized":"2023-07-04T18:51:23.303755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417409+0000","last_change":"2023-07-04T18:51:21.471442+0000","last_active":"2023-07-04T18:51:23.417409+0000","last_peered":"2023-07-04T18:51:23.417409+0000","last_clean":"2023-07-04T18:51:23.417409+0000","last_became_active":"2023-07-04T18:51:21.471130+0000","last_became_peered":"2023-07-04T18:51:21.471130+0000","last_unstale":"2023-07-04T18:51:23.417409+0000","last_undegraded":"2023-07-04T18:51:23.417409+0000","last_fullsized":"2023-07-04T18:51:23.417409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:38.465858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752486+0000","last_change":"2023-07-04T18:51:22.322483+0000","last_active":"2023-07-04T18:51:24.752486+0000","last_peered":"2023-07-04T18:51:24.752486+0000","last_clean":"2023-07-04T18:51:24.752486+0000","last_became_active":"2023-07-04T18:51:22.322262+0000","last_became_peered":"2023-07-04T18:51:22.322262+0000","last_unstale":"2023-07-04T18:51:24.752486+0000","last_undegraded":"2023-07-04T18:51:24.752486+0000","last_fullsized":"2023-07-04T18:51:24.752486+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303402+0000","last_change":"2023-07-04T18:51:21.470525+0000","last_active":"2023-07-04T18:51:23.303402+0000","last_peered":"2023-07-04T18:51:23.303402+0000","last_clean":"2023-07-04T18:51:23.303402+0000","last_became_active":"2023-07-04T18:51:21.470230+0000","last_became_peered":"2023-07-04T18:51:21.470230+0000","last_unstale":"2023-07-04T18:51:23.303402+0000","last_undegraded":"2023-07-04T18:51:23.303402+0000","last_fullsized":"2023-07-04T18:51:23.303402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:44.416126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303729+0000","last_change":"2023-07-04T18:51:22.323982+0000","last_active":"2023-07-04T18:51:23.303729+0000","last_peered":"2023-07-04T18:51:23.303729+0000","last_clean":"2023-07-04T18:51:23.303729+0000","last_became_active":"2023-07-04T18:51:22.323650+0000","last_became_peered":"2023-07-04T18:51:22.323650+0000","last_unstale":"2023-07-04T18:51:23.303729+0000","last_undegraded":"2023-07-04T18:51:23.303729+0000","last_fullsized":"2023-07-04T18:51:23.303729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752611+0000","last_change":"2023-07-04T18:51:21.464672+0000","last_active":"2023-07-04T18:51:24.752611+0000","last_peered":"2023-07-04T18:51:24.752611+0000","last_clean":"2023-07-04T18:51:24.752611+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:24.752611+0000","last_undegraded":"2023-07-04T18:51:24.752611+0000","last_fullsized":"2023-07-04T18:51:24.752611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:09.337120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303369+0000","last_change":"2023-07-04T18:51:22.311834+0000","last_active":"2023-07-04T18:51:23.303369+0000","last_peered":"2023-07-04T18:51:23.303369+0000","last_clean":"2023-07-04T18:51:23.303369+0000","last_became_active":"2023-07-04T18:51:22.311536+0000","last_became_peered":"2023-07-04T18:51:22.311536+0000","last_unstale":"2023-07-04T18:51:23.303369+0000","last_undegraded":"2023-07-04T18:51:23.303369+0000","last_fullsized":"2023-07-04T18:51:23.303369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774759+0000","last_change":"2023-07-04T18:51:21.464692+0000","last_active":"2023-07-04T18:51:24.774759+0000","last_peered":"2023-07-04T18:51:24.774759+0000","last_clean":"2023-07-04T18:51:24.774759+0000","last_became_active":"2023-07-04T18:51:21.464388+0000","last_became_peered":"2023-07-04T18:51:21.464388+0000","last_unstale":"2023-07-04T18:51:24.774759+0000","last_undegraded":"2023-07-04T18:51:24.774759+0000","last_fullsized":"2023-07-04T18:51:24.774759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:52.249980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303533+0000","last_change":"2023-07-04T18:51:22.320250+0000","last_active":"2023-07-04T18:51:23.303533+0000","last_peered":"2023-07-04T18:51:23.303533+0000","last_clean":"2023-07-04T18:51:23.303533+0000","last_became_active":"2023-07-04T18:51:22.319730+0000","last_became_peered":"2023-07-04T18:51:22.319730+0000","last_unstale":"2023-07-04T18:51:23.303533+0000","last_undegraded":"2023-07-04T18:51:23.303533+0000","last_fullsized":"2023-07-04T18:51:23.303533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451104+0000","last_change":"2023-07-04T18:51:21.471723+0000","last_active":"2023-07-04T18:51:23.451104+0000","last_peered":"2023-07-04T18:51:23.451104+0000","last_clean":"2023-07-04T18:51:23.451104+0000","last_became_active":"2023-07-04T18:51:21.471500+0000","last_became_peered":"2023-07-04T18:51:21.471500+0000","last_unstale":"2023-07-04T18:51:23.451104+0000","last_undegraded":"2023-07-04T18:51:23.451104+0000","last_fullsized":"2023-07-04T18:51:23.451104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:14:51.732204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417449+0000","last_change":"2023-07-04T18:51:22.311576+0000","last_active":"2023-07-04T18:51:23.417449+0000","last_peered":"2023-07-04T18:51:23.417449+0000","last_clean":"2023-07-04T18:51:23.417449+0000","last_became_active":"2023-07-04T18:51:22.311212+0000","last_became_peered":"2023-07-04T18:51:22.311212+0000","last_unstale":"2023-07-04T18:51:23.417449+0000","last_undegraded":"2023-07-04T18:51:23.417449+0000","last_fullsized":"2023-07-04T18:51:23.417449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304622+0000","last_change":"2023-07-04T18:51:21.464597+0000","last_active":"2023-07-04T18:51:23.304622+0000","last_peered":"2023-07-04T18:51:23.304622+0000","last_clean":"2023-07-04T18:51:23.304622+0000","last_became_active":"2023-07-04T18:51:21.464263+0000","last_became_peered":"2023-07-04T18:51:21.464263+0000","last_unstale":"2023-07-04T18:51:23.304622+0000","last_undegraded":"2023-07-04T18:51:23.304622+0000","last_fullsized":"2023-07-04T18:51:23.304622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304535+0000","last_change":"2023-07-04T18:51:22.318474+0000","last_active":"2023-07-04T18:51:23.304535+0000","last_peered":"2023-07-04T18:51:23.304535+0000","last_clean":"2023-07-04T18:51:23.304535+0000","last_became_active":"2023-07-04T18:51:22.317476+0000","last_became_peered":"2023-07-04T18:51:22.317476+0000","last_unstale":"2023-07-04T18:51:23.304535+0000","last_undegraded":"2023-07-04T18:51:23.304535+0000","last_fullsized":"2023-07-04T18:51:23.304535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:10.769385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450969+0000","last_change":"2023-07-04T18:51:21.467028+0000","last_active":"2023-07-04T18:51:23.450969+0000","last_peered":"2023-07-04T18:51:23.450969+0000","last_clean":"2023-07-04T18:51:23.450969+0000","last_became_active":"2023-07-04T18:51:21.466697+0000","last_became_peered":"2023-07-04T18:51:21.466697+0000","last_unstale":"2023-07-04T18:51:23.450969+0000","last_undegraded":"2023-07-04T18:51:23.450969+0000","last_fullsized":"2023-07-04T18:51:23.450969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303917+0000","last_change":"2023-07-04T18:51:22.320412+0000","last_active":"2023-07-04T18:51:23.303917+0000","last_peered":"2023-07-04T18:51:23.303917+0000","last_clean":"2023-07-04T18:51:23.303917+0000","last_became_active":"2023-07-04T18:51:22.320022+0000","last_became_peered":"2023-07-04T18:51:22.320022+0000","last_unstale":"2023-07-04T18:51:23.303917+0000","last_undegraded":"2023-07-04T18:51:23.303917+0000","last_fullsized":"2023-07-04T18:51:23.303917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:04.328770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752206+0000","last_change":"2023-07-04T18:51:21.465044+0000","last_active":"2023-07-04T18:51:24.752206+0000","last_peered":"2023-07-04T18:51:24.752206+0000","last_clean":"2023-07-04T18:51:24.752206+0000","last_became_active":"2023-07-04T18:51:21.464683+0000","last_became_peered":"2023-07-04T18:51:21.464683+0000","last_unstale":"2023-07-04T18:51:24.752206+0000","last_undegraded":"2023-07-04T18:51:24.752206+0000","last_fullsized":"2023-07-04T18:51:24.752206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:08:09.767759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774581+0000","last_change":"2023-07-04T18:51:22.324545+0000","last_active":"2023-07-04T18:51:24.774581+0000","last_peered":"2023-07-04T18:51:24.774581+0000","last_clean":"2023-07-04T18:51:24.774581+0000","last_became_active":"2023-07-04T18:51:22.324168+0000","last_became_peered":"2023-07-04T18:51:22.324168+0000","last_unstale":"2023-07-04T18:51:24.774581+0000","last_undegraded":"2023-07-04T18:51:24.774581+0000","last_fullsized":"2023-07-04T18:51:24.774581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:21.786665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450810+0000","last_change":"2023-07-04T18:51:21.468878+0000","last_active":"2023-07-04T18:51:23.450810+0000","last_peered":"2023-07-04T18:51:23.450810+0000","last_clean":"2023-07-04T18:51:23.450810+0000","last_became_active":"2023-07-04T18:51:21.468229+0000","last_became_peered":"2023-07-04T18:51:21.468229+0000","last_unstale":"2023-07-04T18:51:23.450810+0000","last_undegraded":"2023-07-04T18:51:23.450810+0000","last_fullsized":"2023-07-04T18:51:23.450810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752369+0000","last_change":"2023-07-04T18:51:22.317975+0000","last_active":"2023-07-04T18:51:24.752369+0000","last_peered":"2023-07-04T18:51:24.752369+0000","last_clean":"2023-07-04T18:51:24.752369+0000","last_became_active":"2023-07-04T18:51:22.317731+0000","last_became_peered":"2023-07-04T18:51:22.317731+0000","last_unstale":"2023-07-04T18:51:24.752369+0000","last_undegraded":"2023-07-04T18:51:24.752369+0000","last_fullsized":"2023-07-04T18:51:24.752369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303059+0000","last_change":"2023-07-04T18:51:21.460954+0000","last_active":"2023-07-04T18:51:23.303059+0000","last_peered":"2023-07-04T18:51:23.303059+0000","last_clean":"2023-07-04T18:51:23.303059+0000","last_became_active":"2023-07-04T18:51:21.460663+0000","last_became_peered":"2023-07-04T18:51:21.460663+0000","last_unstale":"2023-07-04T18:51:23.303059+0000","last_undegraded":"2023-07-04T18:51:23.303059+0000","last_fullsized":"2023-07-04T18:51:23.303059+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:35:08.443319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304053+0000","last_change":"2023-07-04T18:51:22.313798+0000","last_active":"2023-07-04T18:51:23.304053+0000","last_peered":"2023-07-04T18:51:23.304053+0000","last_clean":"2023-07-04T18:51:23.304053+0000","last_became_active":"2023-07-04T18:51:22.313526+0000","last_became_peered":"2023-07-04T18:51:22.313526+0000","last_unstale":"2023-07-04T18:51:23.304053+0000","last_undegraded":"2023-07-04T18:51:23.304053+0000","last_fullsized":"2023-07-04T18:51:23.304053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451341+0000","last_change":"2023-07-04T18:51:21.468130+0000","last_active":"2023-07-04T18:51:23.451341+0000","last_peered":"2023-07-04T18:51:23.451341+0000","last_clean":"2023-07-04T18:51:23.451341+0000","last_became_active":"2023-07-04T18:51:21.467832+0000","last_became_peered":"2023-07-04T18:51:21.467832+0000","last_unstale":"2023-07-04T18:51:23.451341+0000","last_undegraded":"2023-07-04T18:51:23.451341+0000","last_fullsized":"2023-07-04T18:51:23.451341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:02:58.666739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774284+0000","last_change":"2023-07-04T18:51:22.325399+0000","last_active":"2023-07-04T18:51:24.774284+0000","last_peered":"2023-07-04T18:51:24.774284+0000","last_clean":"2023-07-04T18:51:24.774284+0000","last_became_active":"2023-07-04T18:51:22.324669+0000","last_became_peered":"2023-07-04T18:51:22.324669+0000","last_unstale":"2023-07-04T18:51:24.774284+0000","last_undegraded":"2023-07-04T18:51:24.774284+0000","last_fullsized":"2023-07-04T18:51:24.774284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:40.116887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451202+0000","last_change":"2023-07-04T18:51:21.466954+0000","last_active":"2023-07-04T18:51:23.451202+0000","last_peered":"2023-07-04T18:51:23.451202+0000","last_clean":"2023-07-04T18:51:23.451202+0000","last_became_active":"2023-07-04T18:51:21.466625+0000","last_became_peered":"2023-07-04T18:51:21.466625+0000","last_unstale":"2023-07-04T18:51:23.451202+0000","last_undegraded":"2023-07-04T18:51:23.451202+0000","last_fullsized":"2023-07-04T18:51:23.451202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:17:35.188150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304676+0000","last_change":"2023-07-04T18:51:22.321314+0000","last_active":"2023-07-04T18:51:23.304676+0000","last_peered":"2023-07-04T18:51:23.304676+0000","last_clean":"2023-07-04T18:51:23.304676+0000","last_became_active":"2023-07-04T18:51:22.321008+0000","last_became_peered":"2023-07-04T18:51:22.321008+0000","last_unstale":"2023-07-04T18:51:23.304676+0000","last_undegraded":"2023-07-04T18:51:23.304676+0000","last_fullsized":"2023-07-04T18:51:23.304676+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:48.503216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451276+0000","last_change":"2023-07-04T18:51:21.468101+0000","last_active":"2023-07-04T18:51:23.451276+0000","last_peered":"2023-07-04T18:51:23.451276+0000","last_clean":"2023-07-04T18:51:23.451276+0000","last_became_active":"2023-07-04T18:51:21.467813+0000","last_became_peered":"2023-07-04T18:51:21.467813+0000","last_unstale":"2023-07-04T18:51:23.451276+0000","last_undegraded":"2023-07-04T18:51:23.451276+0000","last_fullsized":"2023-07-04T18:51:23.451276+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:02:10.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303594+0000","last_change":"2023-07-04T18:51:22.324752+0000","last_active":"2023-07-04T18:51:23.303594+0000","last_peered":"2023-07-04T18:51:23.303594+0000","last_clean":"2023-07-04T18:51:23.303594+0000","last_became_active":"2023-07-04T18:51:22.324420+0000","last_became_peered":"2023-07-04T18:51:22.324420+0000","last_unstale":"2023-07-04T18:51:23.303594+0000","last_undegraded":"2023-07-04T18:51:23.303594+0000","last_fullsized":"2023-07-04T18:51:23.303594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:28:06.453079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774466+0000","last_change":"2023-07-04T18:51:21.471602+0000","last_active":"2023-07-04T18:51:24.774466+0000","last_peered":"2023-07-04T18:51:24.774466+0000","last_clean":"2023-07-04T18:51:24.774466+0000","last_became_active":"2023-07-04T18:51:21.471319+0000","last_became_peered":"2023-07-04T18:51:21.471319+0000","last_unstale":"2023-07-04T18:51:24.774466+0000","last_undegraded":"2023-07-04T18:51:24.774466+0000","last_fullsized":"2023-07-04T18:51:24.774466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752233+0000","last_change":"2023-07-04T18:51:22.321087+0000","last_active":"2023-07-04T18:51:24.752233+0000","last_peered":"2023-07-04T18:51:24.752233+0000","last_clean":"2023-07-04T18:51:24.752233+0000","last_became_active":"2023-07-04T18:51:22.320871+0000","last_became_peered":"2023-07-04T18:51:22.320871+0000","last_unstale":"2023-07-04T18:51:24.752233+0000","last_undegraded":"2023-07-04T18:51:24.752233+0000","last_fullsized":"2023-07-04T18:51:24.752233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:50:45.417753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.473477+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.473173+0000","last_became_peered":"2023-07-04T18:51:21.473173+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:49.904135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304599+0000","last_change":"2023-07-04T18:51:22.317634+0000","last_active":"2023-07-04T18:51:23.304599+0000","last_peered":"2023-07-04T18:51:23.304599+0000","last_clean":"2023-07-04T18:51:23.304599+0000","last_became_active":"2023-07-04T18:51:22.317260+0000","last_became_peered":"2023-07-04T18:51:22.317260+0000","last_unstale":"2023-07-04T18:51:23.304599+0000","last_undegraded":"2023-07-04T18:51:23.304599+0000","last_fullsized":"2023-07-04T18:51:23.304599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303472+0000","last_change":"2023-07-04T18:51:21.463128+0000","last_active":"2023-07-04T18:51:23.303472+0000","last_peered":"2023-07-04T18:51:23.303472+0000","last_clean":"2023-07-04T18:51:23.303472+0000","last_became_active":"2023-07-04T18:51:21.462860+0000","last_became_peered":"2023-07-04T18:51:21.462860+0000","last_unstale":"2023-07-04T18:51:23.303472+0000","last_undegraded":"2023-07-04T18:51:23.303472+0000","last_fullsized":"2023-07-04T18:51:23.303472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:15:45.736296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304928+0000","last_change":"2023-07-04T18:51:22.318985+0000","last_active":"2023-07-04T18:51:23.304928+0000","last_peered":"2023-07-04T18:51:23.304928+0000","last_clean":"2023-07-04T18:51:23.304928+0000","last_became_active":"2023-07-04T18:51:22.318393+0000","last_became_peered":"2023-07-04T18:51:22.318393+0000","last_unstale":"2023-07-04T18:51:23.304928+0000","last_undegraded":"2023-07-04T18:51:23.304928+0000","last_fullsized":"2023-07-04T18:51:23.304928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.207247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304517+0000","last_change":"2023-07-04T18:51:21.466769+0000","last_active":"2023-07-04T18:51:23.304517+0000","last_peered":"2023-07-04T18:51:23.304517+0000","last_clean":"2023-07-04T18:51:23.304517+0000","last_became_active":"2023-07-04T18:51:21.466504+0000","last_became_peered":"2023-07-04T18:51:21.466504+0000","last_unstale":"2023-07-04T18:51:23.304517+0000","last_undegraded":"2023-07-04T18:51:23.304517+0000","last_fullsized":"2023-07-04T18:51:23.304517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:59:03.327143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304757+0000","last_change":"2023-07-04T18:51:22.317855+0000","last_active":"2023-07-04T18:51:23.304757+0000","last_peered":"2023-07-04T18:51:23.304757+0000","last_clean":"2023-07-04T18:51:23.304757+0000","last_became_active":"2023-07-04T18:51:22.317426+0000","last_became_peered":"2023-07-04T18:51:22.317426+0000","last_unstale":"2023-07-04T18:51:23.304757+0000","last_undegraded":"2023-07-04T18:51:23.304757+0000","last_fullsized":"2023-07-04T18:51:23.304757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303314+0000","last_change":"2023-07-04T18:51:21.471306+0000","last_active":"2023-07-04T18:51:23.303314+0000","last_peered":"2023-07-04T18:51:23.303314+0000","last_clean":"2023-07-04T18:51:23.303314+0000","last_became_active":"2023-07-04T18:51:21.471034+0000","last_became_peered":"2023-07-04T18:51:21.471034+0000","last_unstale":"2023-07-04T18:51:23.303314+0000","last_undegraded":"2023-07-04T18:51:23.303314+0000","last_fullsized":"2023-07-04T18:51:23.303314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:54.967188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417826+0000","last_change":"2023-07-04T18:51:22.315022+0000","last_active":"2023-07-04T18:51:23.417826+0000","last_peered":"2023-07-04T18:51:23.417826+0000","last_clean":"2023-07-04T18:51:23.417826+0000","last_became_active":"2023-07-04T18:51:22.314767+0000","last_became_peered":"2023-07-04T18:51:22.314767+0000","last_unstale":"2023-07-04T18:51:23.417826+0000","last_undegraded":"2023-07-04T18:51:23.417826+0000","last_fullsized":"2023-07-04T18:51:23.417826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417511+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:51:23.417511+0000","last_peered":"2023-07-04T18:51:23.417511+0000","last_clean":"2023-07-04T18:51:23.417511+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:51:23.417511+0000","last_undegraded":"2023-07-04T18:51:23.417511+0000","last_fullsized":"2023-07-04T18:51:23.417511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774644+0000","last_change":"2023-07-04T18:51:21.471879+0000","last_active":"2023-07-04T18:51:24.774644+0000","last_peered":"2023-07-04T18:51:24.774644+0000","last_clean":"2023-07-04T18:51:24.774644+0000","last_became_active":"2023-07-04T18:51:21.471549+0000","last_became_peered":"2023-07-04T18:51:21.471549+0000","last_unstale":"2023-07-04T18:51:24.774644+0000","last_undegraded":"2023-07-04T18:51:24.774644+0000","last_fullsized":"2023-07-04T18:51:24.774644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:20.930173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417554+0000","last_change":"2023-07-04T18:51:22.319180+0000","last_active":"2023-07-04T18:51:23.417554+0000","last_peered":"2023-07-04T18:51:23.417554+0000","last_clean":"2023-07-04T18:51:23.417554+0000","last_became_active":"2023-07-04T18:51:22.318906+0000","last_became_peered":"2023-07-04T18:51:22.318906+0000","last_unstale":"2023-07-04T18:51:23.417554+0000","last_undegraded":"2023-07-04T18:51:23.417554+0000","last_fullsized":"2023-07-04T18:51:23.417554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:53.826636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774410+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:51:24.774410+0000","last_peered":"2023-07-04T18:51:24.774410+0000","last_clean":"2023-07-04T18:51:24.774410+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:51:24.774410+0000","last_undegraded":"2023-07-04T18:51:24.774410+0000","last_fullsized":"2023-07-04T18:51:24.774410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:48.688640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304959+0000","last_change":"2023-07-04T18:51:21.461532+0000","last_active":"2023-07-04T18:51:23.304959+0000","last_peered":"2023-07-04T18:51:23.304959+0000","last_clean":"2023-07-04T18:51:23.304959+0000","last_became_active":"2023-07-04T18:51:21.460816+0000","last_became_peered":"2023-07-04T18:51:21.460816+0000","last_unstale":"2023-07-04T18:51:23.304959+0000","last_undegraded":"2023-07-04T18:51:23.304959+0000","last_fullsized":"2023-07-04T18:51:23.304959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:53:27.241596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417715+0000","last_change":"2023-07-04T18:51:22.312480+0000","last_active":"2023-07-04T18:51:23.417715+0000","last_peered":"2023-07-04T18:51:23.417715+0000","last_clean":"2023-07-04T18:51:23.417715+0000","last_became_active":"2023-07-04T18:51:22.311193+0000","last_became_peered":"2023-07-04T18:51:22.311193+0000","last_unstale":"2023-07-04T18:51:23.417715+0000","last_undegraded":"2023-07-04T18:51:23.417715+0000","last_fullsized":"2023-07-04T18:51:23.417715+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:11:50.905761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773379+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:51:24.773379+0000","last_peered":"2023-07-04T18:51:24.773379+0000","last_clean":"2023-07-04T18:51:24.773379+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:51:24.773379+0000","last_undegraded":"2023-07-04T18:51:24.773379+0000","last_fullsized":"2023-07-04T18:51:24.773379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:30:15.786775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417530+0000","last_change":"2023-07-04T18:51:21.467449+0000","last_active":"2023-07-04T18:51:23.417530+0000","last_peered":"2023-07-04T18:51:23.417530+0000","last_clean":"2023-07-04T18:51:23.417530+0000","last_became_active":"2023-07-04T18:51:21.466522+0000","last_became_peered":"2023-07-04T18:51:21.466522+0000","last_unstale":"2023-07-04T18:51:23.417530+0000","last_undegraded":"2023-07-04T18:51:23.417530+0000","last_fullsized":"2023-07-04T18:51:23.417530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417692+0000","last_change":"2023-07-04T18:51:22.324258+0000","last_active":"2023-07-04T18:51:23.417692+0000","last_peered":"2023-07-04T18:51:23.417692+0000","last_clean":"2023-07-04T18:51:23.417692+0000","last_became_active":"2023-07-04T18:51:22.323966+0000","last_became_peered":"2023-07-04T18:51:22.323966+0000","last_unstale":"2023-07-04T18:51:23.417692+0000","last_undegraded":"2023-07-04T18:51:23.417692+0000","last_fullsized":"2023-07-04T18:51:23.417692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774412+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:51:24.774412+0000","last_peered":"2023-07-04T18:51:24.774412+0000","last_clean":"2023-07-04T18:51:24.774412+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:51:24.774412+0000","last_undegraded":"2023-07-04T18:51:24.774412+0000","last_fullsized":"2023-07-04T18:51:24.774412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:58:59.190774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":175,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450698+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:51:23.450698+0000","last_peered":"2023-07-04T18:51:23.450698+0000","last_clean":"2023-07-04T18:51:23.450698+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:51:23.450698+0000","last_undegraded":"2023-07-04T18:51:23.450698+0000","last_fullsized":"2023-07-04T18:51:23.450698+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:37.197069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303756+0000","last_change":"2023-07-04T18:51:21.470626+0000","last_active":"2023-07-04T18:51:23.303756+0000","last_peered":"2023-07-04T18:51:23.303756+0000","last_clean":"2023-07-04T18:51:23.303756+0000","last_became_active":"2023-07-04T18:51:21.470351+0000","last_became_peered":"2023-07-04T18:51:21.470351+0000","last_unstale":"2023-07-04T18:51:23.303756+0000","last_undegraded":"2023-07-04T18:51:23.303756+0000","last_fullsized":"2023-07-04T18:51:23.303756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:09:39.115261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303308+0000","last_change":"2023-07-04T18:51:22.321449+0000","last_active":"2023-07-04T18:51:23.303308+0000","last_peered":"2023-07-04T18:51:23.303308+0000","last_clean":"2023-07-04T18:51:23.303308+0000","last_became_active":"2023-07-04T18:51:22.321158+0000","last_became_peered":"2023-07-04T18:51:22.321158+0000","last_unstale":"2023-07-04T18:51:23.303308+0000","last_undegraded":"2023-07-04T18:51:23.303308+0000","last_fullsized":"2023-07-04T18:51:23.303308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:36:44.921156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.450897+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:51:23.450897+0000","last_peered":"2023-07-04T18:51:23.450897+0000","last_clean":"2023-07-04T18:51:23.450897+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:51:23.450897+0000","last_undegraded":"2023-07-04T18:51:23.450897+0000","last_fullsized":"2023-07-04T18:51:23.450897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:35:48.113610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752148+0000","last_change":"2023-07-04T18:51:21.460314+0000","last_active":"2023-07-04T18:51:24.752148+0000","last_peered":"2023-07-04T18:51:24.752148+0000","last_clean":"2023-07-04T18:51:24.752148+0000","last_became_active":"2023-07-04T18:51:21.460022+0000","last_became_peered":"2023-07-04T18:51:21.460022+0000","last_unstale":"2023-07-04T18:51:24.752148+0000","last_undegraded":"2023-07-04T18:51:24.752148+0000","last_fullsized":"2023-07-04T18:51:24.752148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:50.578446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752641+0000","last_change":"2023-07-04T18:51:22.321571+0000","last_active":"2023-07-04T18:51:24.752641+0000","last_peered":"2023-07-04T18:51:24.752641+0000","last_clean":"2023-07-04T18:51:24.752641+0000","last_became_active":"2023-07-04T18:51:22.321326+0000","last_became_peered":"2023-07-04T18:51:22.321326+0000","last_unstale":"2023-07-04T18:51:24.752641+0000","last_undegraded":"2023-07-04T18:51:24.752641+0000","last_fullsized":"2023-07-04T18:51:24.752641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:07.675529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.451151+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:51:23.451151+0000","last_peered":"2023-07-04T18:51:23.451151+0000","last_clean":"2023-07-04T18:51:23.451151+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:51:23.451151+0000","last_undegraded":"2023-07-04T18:51:23.451151+0000","last_fullsized":"2023-07-04T18:51:23.451151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:14:08.639700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417718+0000","last_change":"2023-07-04T18:51:21.466845+0000","last_active":"2023-07-04T18:51:23.417718+0000","last_peered":"2023-07-04T18:51:23.417718+0000","last_clean":"2023-07-04T18:51:23.417718+0000","last_became_active":"2023-07-04T18:51:21.466546+0000","last_became_peered":"2023-07-04T18:51:21.466546+0000","last_unstale":"2023-07-04T18:51:23.417718+0000","last_undegraded":"2023-07-04T18:51:23.417718+0000","last_fullsized":"2023-07-04T18:51:23.417718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752539+0000","last_change":"2023-07-04T18:51:22.318511+0000","last_active":"2023-07-04T18:51:24.752539+0000","last_peered":"2023-07-04T18:51:24.752539+0000","last_clean":"2023-07-04T18:51:24.752539+0000","last_became_active":"2023-07-04T18:51:22.318227+0000","last_became_peered":"2023-07-04T18:51:22.318227+0000","last_unstale":"2023-07-04T18:51:24.752539+0000","last_undegraded":"2023-07-04T18:51:24.752539+0000","last_fullsized":"2023-07-04T18:51:24.752539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:14.173575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304031+0000","last_change":"2023-07-04T18:51:21.474661+0000","last_active":"2023-07-04T18:51:23.304031+0000","last_peered":"2023-07-04T18:51:23.304031+0000","last_clean":"2023-07-04T18:51:23.304031+0000","last_became_active":"2023-07-04T18:51:21.474400+0000","last_became_peered":"2023-07-04T18:51:21.474400+0000","last_unstale":"2023-07-04T18:51:23.304031+0000","last_undegraded":"2023-07-04T18:51:23.304031+0000","last_fullsized":"2023-07-04T18:51:23.304031+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417697+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:51:23.417697+0000","last_peered":"2023-07-04T18:51:23.417697+0000","last_clean":"2023-07-04T18:51:23.417697+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:51:23.417697+0000","last_undegraded":"2023-07-04T18:51:23.417697+0000","last_fullsized":"2023-07-04T18:51:23.417697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:53:02.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303243+0000","last_change":"2023-07-04T18:51:22.313754+0000","last_active":"2023-07-04T18:51:23.303243+0000","last_peered":"2023-07-04T18:51:23.303243+0000","last_clean":"2023-07-04T18:51:23.303243+0000","last_became_active":"2023-07-04T18:51:22.313509+0000","last_became_peered":"2023-07-04T18:51:22.313509+0000","last_unstale":"2023-07-04T18:51:23.303243+0000","last_undegraded":"2023-07-04T18:51:23.303243+0000","last_fullsized":"2023-07-04T18:51:23.303243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:30.653867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303419+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:51:23.303419+0000","last_peered":"2023-07-04T18:51:23.303419+0000","last_clean":"2023-07-04T18:51:23.303419+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:51:23.303419+0000","last_undegraded":"2023-07-04T18:51:23.303419+0000","last_fullsized":"2023-07-04T18:51:23.303419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:49:53.562592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752666+0000","last_change":"2023-07-04T18:51:21.464739+0000","last_active":"2023-07-04T18:51:24.752666+0000","last_peered":"2023-07-04T18:51:24.752666+0000","last_clean":"2023-07-04T18:51:24.752666+0000","last_became_active":"2023-07-04T18:51:21.464445+0000","last_became_peered":"2023-07-04T18:51:21.464445+0000","last_unstale":"2023-07-04T18:51:24.752666+0000","last_undegraded":"2023-07-04T18:51:24.752666+0000","last_fullsized":"2023-07-04T18:51:24.752666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:20.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304372+0000","last_change":"2023-07-04T18:51:22.326679+0000","last_active":"2023-07-04T18:51:23.304372+0000","last_peered":"2023-07-04T18:51:23.304372+0000","last_clean":"2023-07-04T18:51:23.304372+0000","last_became_active":"2023-07-04T18:51:22.326348+0000","last_became_peered":"2023-07-04T18:51:22.326348+0000","last_unstale":"2023-07-04T18:51:23.304372+0000","last_undegraded":"2023-07-04T18:51:23.304372+0000","last_fullsized":"2023-07-04T18:51:23.304372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:39.467484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752305+0000","last_change":"2023-07-04T18:51:21.466007+0000","last_active":"2023-07-04T18:51:24.752305+0000","last_peered":"2023-07-04T18:51:24.752305+0000","last_clean":"2023-07-04T18:51:24.752305+0000","last_became_active":"2023-07-04T18:51:21.465754+0000","last_became_peered":"2023-07-04T18:51:21.465754+0000","last_unstale":"2023-07-04T18:51:24.752305+0000","last_undegraded":"2023-07-04T18:51:24.752305+0000","last_fullsized":"2023-07-04T18:51:24.752305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:17.208673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.773596+0000","last_change":"2023-07-04T18:51:22.309513+0000","last_active":"2023-07-04T18:51:24.773596+0000","last_peered":"2023-07-04T18:51:24.773596+0000","last_clean":"2023-07-04T18:51:24.773596+0000","last_became_active":"2023-07-04T18:51:22.309252+0000","last_became_peered":"2023-07-04T18:51:22.309252+0000","last_unstale":"2023-07-04T18:51:24.773596+0000","last_undegraded":"2023-07-04T18:51:24.773596+0000","last_fullsized":"2023-07-04T18:51:24.773596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774594+0000","last_change":"2023-07-04T18:51:21.463261+0000","last_active":"2023-07-04T18:51:24.774594+0000","last_peered":"2023-07-04T18:51:24.774594+0000","last_clean":"2023-07-04T18:51:24.774594+0000","last_became_active":"2023-07-04T18:51:21.462985+0000","last_became_peered":"2023-07-04T18:51:21.462985+0000","last_unstale":"2023-07-04T18:51:24.774594+0000","last_undegraded":"2023-07-04T18:51:24.774594+0000","last_fullsized":"2023-07-04T18:51:24.774594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:47.501589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304859+0000","last_change":"2023-07-04T18:51:22.318260+0000","last_active":"2023-07-04T18:51:23.304859+0000","last_peered":"2023-07-04T18:51:23.304859+0000","last_clean":"2023-07-04T18:51:23.304859+0000","last_became_active":"2023-07-04T18:51:22.317037+0000","last_became_peered":"2023-07-04T18:51:22.317037+0000","last_unstale":"2023-07-04T18:51:23.304859+0000","last_undegraded":"2023-07-04T18:51:23.304859+0000","last_fullsized":"2023-07-04T18:51:23.304859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:13:21.931800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.774762+0000","last_change":"2023-07-04T18:51:21.463234+0000","last_active":"2023-07-04T18:51:24.774762+0000","last_peered":"2023-07-04T18:51:24.774762+0000","last_clean":"2023-07-04T18:51:24.774762+0000","last_became_active":"2023-07-04T18:51:21.462954+0000","last_became_peered":"2023-07-04T18:51:21.462954+0000","last_unstale":"2023-07-04T18:51:24.774762+0000","last_undegraded":"2023-07-04T18:51:24.774762+0000","last_fullsized":"2023-07-04T18:51:24.774762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:56:52.205620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304612+0000","last_change":"2023-07-04T18:51:22.317628+0000","last_active":"2023-07-04T18:51:23.304612+0000","last_peered":"2023-07-04T18:51:23.304612+0000","last_clean":"2023-07-04T18:51:23.304612+0000","last_became_active":"2023-07-04T18:51:22.317043+0000","last_became_peered":"2023-07-04T18:51:22.317043+0000","last_unstale":"2023-07-04T18:51:23.304612+0000","last_undegraded":"2023-07-04T18:51:23.304612+0000","last_fullsized":"2023-07-04T18:51:23.304612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303923+0000","last_change":"2023-07-04T18:51:21.470557+0000","last_active":"2023-07-04T18:51:23.303923+0000","last_peered":"2023-07-04T18:51:23.303923+0000","last_clean":"2023-07-04T18:51:23.303923+0000","last_became_active":"2023-07-04T18:51:21.470247+0000","last_became_peered":"2023-07-04T18:51:21.470247+0000","last_unstale":"2023-07-04T18:51:23.303923+0000","last_undegraded":"2023-07-04T18:51:23.303923+0000","last_fullsized":"2023-07-04T18:51:23.303923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:23:13.171949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303469+0000","last_change":"2023-07-04T18:51:22.313146+0000","last_active":"2023-07-04T18:51:23.303469+0000","last_peered":"2023-07-04T18:51:23.303469+0000","last_clean":"2023-07-04T18:51:23.303469+0000","last_became_active":"2023-07-04T18:51:22.312877+0000","last_became_peered":"2023-07-04T18:51:22.312877+0000","last_unstale":"2023-07-04T18:51:23.303469+0000","last_undegraded":"2023-07-04T18:51:23.303469+0000","last_fullsized":"2023-07-04T18:51:23.303469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:24.752619+0000","last_change":"2023-07-04T18:51:21.459772+0000","last_active":"2023-07-04T18:51:24.752619+0000","last_peered":"2023-07-04T18:51:24.752619+0000","last_clean":"2023-07-04T18:51:24.752619+0000","last_became_active":"2023-07-04T18:51:21.459459+0000","last_became_peered":"2023-07-04T18:51:21.459459+0000","last_unstale":"2023-07-04T18:51:24.752619+0000","last_undegraded":"2023-07-04T18:51:24.752619+0000","last_fullsized":"2023-07-04T18:51:24.752619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:20.440921+0000","last_clean_scrub_stamp":"2023-07-04T18:51:20.440921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:03:47.501589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303922+0000","last_change":"2023-07-04T18:51:22.318229+0000","last_active":"2023-07-04T18:51:23.303922+0000","last_peered":"2023-07-04T18:51:23.303922+0000","last_clean":"2023-07-04T18:51:23.303922+0000","last_became_active":"2023-07-04T18:51:22.317388+0000","last_became_peered":"2023-07-04T18:51:22.317388+0000","last_unstale":"2023-07-04T18:51:23.303922+0000","last_undegraded":"2023-07-04T18:51:23.303922+0000","last_fullsized":"2023-07-04T18:51:23.303922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:22:55.968814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303155+0000","last_change":"2023-07-04T18:51:22.314933+0000","last_active":"2023-07-04T18:51:23.303155+0000","last_peered":"2023-07-04T18:51:23.303155+0000","last_clean":"2023-07-04T18:51:23.303155+0000","last_became_active":"2023-07-04T18:51:22.314648+0000","last_became_peered":"2023-07-04T18:51:22.314648+0000","last_unstale":"2023-07-04T18:51:23.303155+0000","last_undegraded":"2023-07-04T18:51:23.303155+0000","last_fullsized":"2023-07-04T18:51:23.303155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:50:42.486433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303365+0000","last_change":"2023-07-04T18:51:22.318422+0000","last_active":"2023-07-04T18:51:23.303365+0000","last_peered":"2023-07-04T18:51:23.303365+0000","last_clean":"2023-07-04T18:51:23.303365+0000","last_became_active":"2023-07-04T18:51:22.318161+0000","last_became_peered":"2023-07-04T18:51:22.318161+0000","last_unstale":"2023-07-04T18:51:23.303365+0000","last_undegraded":"2023-07-04T18:51:23.303365+0000","last_fullsized":"2023-07-04T18:51:23.303365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:34:51.580072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.304314+0000","last_change":"2023-07-04T18:51:22.320144+0000","last_active":"2023-07-04T18:51:23.304314+0000","last_peered":"2023-07-04T18:51:23.304314+0000","last_clean":"2023-07-04T18:51:23.304314+0000","last_became_active":"2023-07-04T18:51:22.317892+0000","last_became_peered":"2023-07-04T18:51:22.317892+0000","last_unstale":"2023-07-04T18:51:23.304314+0000","last_undegraded":"2023-07-04T18:51:23.304314+0000","last_fullsized":"2023-07-04T18:51:23.304314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:01:18.210299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.303934+0000","last_change":"2023-07-04T18:51:22.317154+0000","last_active":"2023-07-04T18:51:23.303934+0000","last_peered":"2023-07-04T18:51:23.303934+0000","last_clean":"2023-07-04T18:51:23.303934+0000","last_became_active":"2023-07-04T18:51:22.316916+0000","last_became_peered":"2023-07-04T18:51:22.316916+0000","last_unstale":"2023-07-04T18:51:23.303934+0000","last_undegraded":"2023-07-04T18:51:23.303934+0000","last_fullsized":"2023-07-04T18:51:23.303934+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:25:10.338747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-07-04T18:51:23.417430+0000","last_change":"2023-07-04T18:51:22.312307+0000","last_active":"2023-07-04T18:51:23.417430+0000","last_peered":"2023-07-04T18:51:23.417430+0000","last_clean":"2023-07-04T18:51:23.417430+0000","last_became_active":"2023-07-04T18:51:22.310832+0000","last_became_peered":"2023-07-04T18:51:22.310832+0000","last_unstale":"2023-07-04T18:51:23.417430+0000","last_undegraded":"2023-07-04T18:51:23.417430+0000","last_fullsized":"2023-07-04T18:51:23.417430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:21.442547+0000","last_clean_scrub_stamp":"2023-07-04T18:51:21.442547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:35:10.352699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":864256,"data_stored":681747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.7719999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.742}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":262144,"data_stored":91219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":249856,"data_stored":90251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:51:31.609 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-07-04T18:51:31.610 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-04T18:51:31.610 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-07-04T18:51:31.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-07-04T18:51:31.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 Processor -- start 2023-07-04T18:51:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 -- start start 2023-07-04T18:51:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 0x7f750c0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f750c06c760 0x7f750c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c069e40 0x7f750c06a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0b2ee0 con 0x7f750c06c760 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0b4a20 con 0x7f750c15c640 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f7512a16700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c06a800 con 0x7f750c069e40 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.762+0000 7f750b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f750c06c760 0x7f750c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f750c06c760 0x7f750c0b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44376/0 (socket says 172.21.15.77:44376) 2023-07-04T18:51:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 -- 172.21.15.77:0/2959866371 learned_addr learned my addr 172.21.15.77:0/2959866371 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750bfff700 1 --2- 172.21.15.77:0/2959866371 >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 0x7f750c0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f7510fb3700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c069e40 0x7f750c06a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 -- 172.21.15.77:0/2959866371 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c069e40 msgr2=0x7f750c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c069e40 0x7f750c06a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 -- 172.21.15.77:0/2959866371 >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 msgr2=0x7f750c0710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 --2- 172.21.15.77:0/2959866371 >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 0x7f750c0710a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 -- 172.21.15.77:0/2959866371 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0b4f10 con 0x7f750c06c760 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750bfff700 1 --2- 172.21.15.77:0/2959866371 >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 0x7f750c0710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750b7fe700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c0b3300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f750c070230 tx=0x7f7500009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbd3e79e3aedc40 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750affd700 1 -- 172.21.15.77:0/2959866371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000b870 con 0x7f750c06c760 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750affd700 1 -- 172.21.15.77:0/2959866371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000b9d0 con 0x7f750c06c760 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.763+0000 7f750affd700 1 -- 172.21.15.77:0/2959866371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000bd60 con 0x7f750c06c760 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 -- 172.21.15.77:0/2959866371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 msgr2=0x7f750c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c0b3300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f750c070230 tx=0x7f7500009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 -- 172.21.15.77:0/2959866371 shutdown_connections 2023-07-04T18:51:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c069e40 0x7f750c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 --2- 172.21.15.77:0/2959866371 >> v2:172.21.15.82:3300/0 conn(0x7f750c15c640 0x7f750c0710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 --2- 172.21.15.77:0/2959866371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c0b3300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 -- 172.21.15.77:0/2959866371 >> 172.21.15.77:0/2959866371 conn(0x7f750c06bb70 msgr2=0x7f750c0b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 -- 172.21.15.77:0/2959866371 shutdown_connections 2023-07-04T18:51:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.764+0000 7f7512a16700 1 -- 172.21.15.77:0/2959866371 wait complete. 2023-07-04T18:51:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.765+0000 7f7512a16700 1 Processor -- start 2023-07-04T18:51:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.765+0000 7f7512a16700 1 -- start start 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f750c069e40 0x7f750c151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c15c640 0x7f750c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1f2250 con 0x7f750c06c760 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1f2530 con 0x7f750c069e40 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7512a16700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1f2810 con 0x7f750c15c640 2023-07-04T18:51:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750bfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f750c069e40 0x7f750c151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44382/0 (socket says 172.21.15.77:44382) 2023-07-04T18:51:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 -- 172.21.15.77:0/3223755356 learned_addr learned my addr 172.21.15.77:0/3223755356 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f7510fb3700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c15c640 0x7f750c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 -- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c15c640 msgr2=0x7f750c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c15c640 0x7f750c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 -- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f750c069e40 msgr2=0x7f750c151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f750c069e40 0x7f750c151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 -- 172.21.15.77:0/3223755356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009550 con 0x7f750c06c760 2023-07-04T18:51:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.766+0000 7f750b7fe700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7500000c00 tx=0x7f7500004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.767+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500004310 con 0x7f750c06c760 2023-07-04T18:51:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.767+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750001e070 con 0x7f750c06c760 2023-07-04T18:51:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.767+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c1f2af0 con 0x7f750c06c760 2023-07-04T18:51:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.767+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75000036a0 con 0x7f750c06c760 2023-07-04T18:51:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.767+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c1537f0 con 0x7f750c06c760 2023-07-04T18:51:31.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.768+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c0af880 con 0x7f750c06c760 2023-07-04T18:51:31.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.768+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7500020030 con 0x7f750c06c760 2023-07-04T18:51:31.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.768+0000 7f7508ff9700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 0x7f74f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:31.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.768+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f750008a2d0 con 0x7f750c06c760 2023-07-04T18:51:31.782 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.769+0000 7f750bfff700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 0x7f74f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:31.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.769+0000 7f750bfff700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 0x7f74f806d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00c940 tx=0x7f74fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:31.783 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:31.773+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f750c0af880 con 0x7f750c06c760 2023-07-04T18:51:32.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.002+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f750c0af880 con 0x7f750c06c760 2023-07-04T18:51:32.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.002+0000 7f7508ff9700 1 -- 172.21.15.77:0/3223755356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f750c0af880 con 0x7f750c06c760 2023-07-04T18:51:32.004 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:32.005 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-04T18:51:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 msgr2=0x7f74f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 0x7f74f806d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00c940 tx=0x7f74fc008040 comp rx=0 tx=0).stop 2023-07-04T18:51:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 msgr2=0x7f750c146e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7500000c00 tx=0x7f7500004750 comp rx=0 tx=0).stop 2023-07-04T18:51:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 shutdown_connections 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f74f806ace0 0x7f74f806d190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f750c15c640 0x7f750c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f750c069e40 0x7f750c151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 --2- 172.21.15.77:0/3223755356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f750c06c760 0x7f750c146e70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 >> 172.21.15.77:0/3223755356 conn(0x7f750c06bb70 msgr2=0x7f750c0d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 shutdown_connections 2023-07-04T18:51:32.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.004+0000 7f7512a16700 1 -- 172.21.15.77:0/3223755356 wait complete. 2023-07-04T18:51:32.018 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-07-04T18:51:32.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.136+0000 7fa129962700 1 Processor -- start 2023-07-04T18:51:32.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.136+0000 7fa129962700 1 -- start start 2023-07-04T18:51:32.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa129962700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240d33c0 0x7fa1240b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa129962700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1240af8f0 0x7fa1240b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa129962700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1240b2790 con 0x7fa1240b1860 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa129962700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1240b2a70 con 0x7fa1240af8f0 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa129962700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240b2d50 con 0x7fa1240d33c0 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa1240af8f0 0x7fa1240b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa122ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa1240b1860 0x7fa1240d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51526/0 (socket says 172.21.15.77:51526) 2023-07-04T18:51:32.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa122ffd700 1 -- 172.21.15.77:0/720631190 learned_addr learned my addr 172.21.15.77:0/720631190 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:32.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1227fc700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240d33c0 0x7fa1240b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 -- 172.21.15.77:0/720631190 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240d33c0 msgr2=0x7fa1240b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240d33c0 0x7fa1240b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 -- 172.21.15.77:0/720631190 >> v1:172.21.15.77:6789/0 conn(0x7fa1240b1860 legacy=0x7fa1240d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa121ffb700 1 -- 172.21.15.77:0/720631190 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033897825 0 0) 0x7fa1240b2790 con 0x7fa1240b1860 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 -- 172.21.15.77:0/720631190 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1240b3030 con 0x7fa1240af8f0 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.137+0000 7fa1237fe700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa1240b3d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009d90 tx=0x7fa11c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7f9003d99570db server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa121ffb700 1 -- 172.21.15.77:0/720631190 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00e070 con 0x7fa1240af8f0 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa121ffb700 1 -- 172.21.15.77:0/720631190 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c00bb20 con 0x7fa1240af8f0 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa121ffb700 1 -- 172.21.15.77:0/720631190 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c012430 con 0x7fa1240af8f0 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 -- 172.21.15.77:0/720631190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 msgr2=0x7fa1240b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa1240b3d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009d90 tx=0x7fa11c00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 -- 172.21.15.77:0/720631190 shutdown_connections 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa1240b3d70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 --2- 172.21.15.77:0/720631190 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240d33c0 0x7fa1240b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 -- 172.21.15.77:0/720631190 >> 172.21.15.77:0/720631190 conn(0x7fa124071e90 msgr2=0x7fa12409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 -- 172.21.15.77:0/720631190 shutdown_connections 2023-07-04T18:51:32.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.138+0000 7fa129962700 1 -- 172.21.15.77:0/720631190 wait complete. 2023-07-04T18:51:32.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.139+0000 7fa129962700 1 Processor -- start 2023-07-04T18:51:32.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.139+0000 7fa129962700 1 -- start start 2023-07-04T18:51:32.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240b1860 0x7fa1240c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1240d33c0 0x7fa124159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c92f0 con 0x7fa1240af8f0 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c83f0 con 0x7fa1240d33c0 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa129962700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240c8670 con 0x7fa1240b1860 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa122ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa122ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44394/0 (socket says 172.21.15.77:44394) 2023-07-04T18:51:32.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa122ffd700 1 -- 172.21.15.77:0/921065230 learned_addr learned my addr 172.21.15.77:0/921065230 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa1227fc700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240b1860 0x7fa1240c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.140+0000 7fa1237fe700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1240d33c0 0x7fa124159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 -- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240b1860 msgr2=0x7fa1240c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240b1860 0x7fa1240c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 -- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1240d33c0 msgr2=0x7fa124159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1240d33c0 0x7fa124159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 -- 172.21.15.77:0/921065230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c009a70 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa122ffd700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1180011e0 tx=0x7fa11800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118004c80 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa118004e00 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1240c8950 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118007310 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.141+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1240c8f10 con 0x7fa1240af8f0 2023-07-04T18:51:32.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.142+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12404e210 con 0x7fa1240af8f0 2023-07-04T18:51:32.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.142+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa118007470 con 0x7fa1240af8f0 2023-07-04T18:51:32.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.143+0000 7fa10bfff700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 0x7fa11006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.143+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa118089040 con 0x7fa1240af8f0 2023-07-04T18:51:32.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.143+0000 7fa1227fc700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 0x7fa11006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.143+0000 7fa1227fc700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 0x7fa11006d190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa114009b40 tx=0x7fa114009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.152 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.147+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa118054180 con 0x7fa1240af8f0 2023-07-04T18:51:32.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.336+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1240b24d0 con 0x7fa1240af8f0 2023-07-04T18:51:32.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.337+0000 7fa10bfff700 1 -- 172.21.15.77:0/921065230 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7fa1180577a0 con 0x7fa1240af8f0 2023-07-04T18:51:32.339 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:32.340 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:32.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 msgr2=0x7fa11006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 0x7fa11006d190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa114009b40 tx=0x7fa114009210 comp rx=0 tx=0).stop 2023-07-04T18:51:32.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 msgr2=0x7fa124159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1180011e0 tx=0x7fa11800ca30 comp rx=0 tx=0).stop 2023-07-04T18:51:32.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 shutdown_connections 2023-07-04T18:51:32.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa11006ace0 0x7fa11006d190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1240af8f0 0x7fa124159ec0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1240d33c0 0x7fa124159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 --2- 172.21.15.77:0/921065230 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa1240b1860 0x7fa1240c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 >> 172.21.15.77:0/921065230 conn(0x7fa124071e90 msgr2=0x7fa1240b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 shutdown_connections 2023-07-04T18:51:32.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.340+0000 7fa129962700 1 -- 172.21.15.77:0/921065230 wait complete. 2023-07-04T18:51:32.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:32.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:32.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.476+0000 7fa377bde700 1 Processor -- start 2023-07-04T18:51:32.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.476+0000 7fa377bde700 1 -- start start 2023-07-04T18:51:32.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa377bde700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa37015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa377bde700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fa3700b3330 0x7fa3700b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa377bde700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3700b3f60 con 0x7fa37006cfc0 2023-07-04T18:51:32.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa377bde700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37006caa0 con 0x7fa3700b3330 2023-07-04T18:51:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa377bde700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37015dc10 con 0x7fa3701590b0 2023-07-04T18:51:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa37617b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa37006cfc0 0x7fa370159480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51542/0 (socket says 172.21.15.77:51542) 2023-07-04T18:51:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa37617b700 1 -- 172.21.15.77:0/963662875 learned_addr learned my addr 172.21.15.77:0/963662875 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:32.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa375179700 1 --2- 172.21.15.77:0/963662875 >> v2:172.21.15.82:3300/0 conn(0x7fa3700b3330 0x7fa3700b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098288421 0 0) 0x7fa3700b3f60 con 0x7fa37006cfc0 2023-07-04T18:51:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa37597a700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa37015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.477+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c0035b0 con 0x7fa37006cfc0 2023-07-04T18:51:32.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609965001 0 0) 0x7fa35c0035b0 con 0x7fa37006cfc0 2023-07-04T18:51:32.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 msgr2=0x7fa37015d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa37015d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 >> v2:172.21.15.82:3300/0 conn(0x7fa3700b3330 msgr2=0x7fa3700b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 --2- 172.21.15.77:0/963662875 >> v2:172.21.15.82:3300/0 conn(0x7fa3700b3330 0x7fa3700b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37006c4e0 con 0x7fa37006cfc0 2023-07-04T18:51:32.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fa3680019f0 con 0x7fa37006cfc0 2023-07-04T18:51:32.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 >> v1:172.21.15.77:6789/0 conn(0x7fa37006cfc0 legacy=0x7fa370159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa35c003a00 0x7fa35c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa35c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa35c004d90 0x7fa35c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37006caa0 con 0x7fa35c003a00 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37015dc10 con 0x7fa35c004d90 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c00ba10 con 0x7fa3701590b0 2023-07-04T18:51:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa35c003a00 0x7fa35c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa375179700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa35c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37617b700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa35c004d90 0x7fa35c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 -- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 msgr2=0x7fa35c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3701590b0 0x7fa35c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 -- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa35c004d90 msgr2=0x7fa35c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa35c004d90 0x7fa35c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 -- 172.21.15.77:0/963662875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0035b0 con 0x7fa35c003a00 2023-07-04T18:51:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37617b700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa35c004d90 0x7fa35c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa37597a700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa35c003a00 0x7fa35c003e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009b60 tx=0x7fa35c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e1a6c12191ad1b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c00e6c0 con 0x7fa35c003a00 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c00e820 con 0x7fa35c003a00 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.478+0000 7fa374978700 1 -- 172.21.15.77:0/963662875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c00eb90 con 0x7fa35c003a00 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 -- 172.21.15.77:0/963662875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa35c003a00 msgr2=0x7fa35c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 --2- 172.21.15.77:0/963662875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa35c003a00 0x7fa35c003e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009b60 tx=0x7fa35c005880 comp rx=0 tx=0).stop 2023-07-04T18:51:32.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa37617b700 1 -- 172.21.15.77:0/963662875 reap_dead start 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 -- 172.21.15.77:0/963662875 shutdown_connections 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 -- 172.21.15.77:0/963662875 >> 172.21.15.77:0/963662875 conn(0x7fa37006af50 msgr2=0x7fa37006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 -- 172.21.15.77:0/963662875 shutdown_connections 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.479+0000 7fa377bde700 1 -- 172.21.15.77:0/963662875 wait complete. 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.480+0000 7fa377bde700 1 Processor -- start 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.480+0000 7fa377bde700 1 -- start start 2023-07-04T18:51:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 0x7fa370142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa370143b30 0x7fa370143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa370143570 con 0x7fa3700b1560 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3701444e0 con 0x7fa370143b30 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa377bde700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa370142750 con 0x7fa35c003a00 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa37597a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 0x7fa370142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa37597a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 0x7fa370142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34262/0 (socket says 172.21.15.77:34262) 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44414/0 (socket says 172.21.15.77:44414) 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa37597a700 1 -- 172.21.15.77:0/3999752249 learned_addr learned my addr 172.21.15.77:0/3999752249 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa37617b700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa370143b30 0x7fa370143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 -- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 msgr2=0x7fa370142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 0x7fa370142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 -- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa370143b30 msgr2=0x7fa370143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa370143b30 0x7fa370143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 -- 172.21.15.77:0/3999752249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3701429f0 con 0x7fa3700b1560 2023-07-04T18:51:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.481+0000 7fa375179700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa360011980 tx=0x7fa360011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.482+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa360004030 con 0x7fa3700b1560 2023-07-04T18:51:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.482+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa360004190 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.482+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa370142cd0 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.482+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa360004500 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.482+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa370143290 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.483+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37004e210 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.483+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa36000f730 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.483+0000 7fa3667fc700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 0x7fa35406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.483+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa360089700 con 0x7fa3700b1560 2023-07-04T18:51:32.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.484+0000 7fa37597a700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 0x7fa35406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.484+0000 7fa37597a700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 0x7fa35406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00a010 tx=0x7fa36c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.488+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa360054840 con 0x7fa3700b1560 2023-07-04T18:51:32.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.674+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3700af560 con 0x7fa3700b1560 2023-07-04T18:51:32.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.674+0000 7fa3667fc700 1 -- 172.21.15.77:0/3999752249 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa360006cc0 con 0x7fa3700b1560 2023-07-04T18:51:32.677 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:32.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 msgr2=0x7fa35406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 0x7fa35406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00a010 tx=0x7fa36c008030 comp rx=0 tx=0).stop 2023-07-04T18:51:32.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 msgr2=0x7fa37013e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa360011980 tx=0x7fa360011c90 comp rx=0 tx=0).stop 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 shutdown_connections 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa35406ace0 0x7fa35406d190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa370143b30 0x7fa370143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa35c003a00 0x7fa370142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 --2- 172.21.15.77:0/3999752249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3700b1560 0x7fa37013e3e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 >> 172.21.15.77:0/3999752249 conn(0x7fa37006af50 msgr2=0x7fa37015b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 shutdown_connections 2023-07-04T18:51:32.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.676+0000 7fa377bde700 1 -- 172.21.15.77:0/3999752249 wait complete. 2023-07-04T18:51:32.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:32.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.811+0000 7f394f51c700 1 Processor -- start 2023-07-04T18:51:32.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 -- start start 2023-07-04T18:51:32.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f394806ea90 0x7f394815d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3948160a80 0x7f3948069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394815da80 con 0x7f394806a380 2023-07-04T18:51:32.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394806a800 con 0x7f3948160a80 2023-07-04T18:51:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394f51c700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480ac8a0 con 0x7f394806ea90 2023-07-04T18:51:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.812+0000 7f394cab7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3948160a80 0x7f3948069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3948160a80 0x7f3948069e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44420/0 (socket says 172.21.15.77:44420) 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 -- 172.21.15.77:0/228981921 learned_addr learned my addr 172.21.15.77:0/228981921 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 -- 172.21.15.77:0/228981921 >> v2:172.21.15.82:3300/0 conn(0x7f394806ea90 msgr2=0x7f394815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 --2- 172.21.15.77:0/228981921 >> v2:172.21.15.82:3300/0 conn(0x7f394806ea90 0x7f394815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 -- 172.21.15.77:0/228981921 >> v1:172.21.15.82:6789/0 conn(0x7f394806a380 legacy=0x7f39480b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 -- 172.21.15.77:0/228981921 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394815d0b0 con 0x7f3948160a80 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394cab7700 1 --2- 172.21.15.77:0/228981921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3948160a80 0x7f3948069e40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f394806ab70 tx=0x7f3944009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a4e36d040e4f89 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f393ffff700 1 -- 172.21.15.77:0/228981921 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944004230 con 0x7f3948160a80 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f393ffff700 1 -- 172.21.15.77:0/228981921 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3944004390 con 0x7f3948160a80 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f393ffff700 1 -- 172.21.15.77:0/228981921 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39440036a0 con 0x7f3948160a80 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.813+0000 7f394f51c700 1 -- 172.21.15.77:0/228981921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3948160a80 msgr2=0x7f3948069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 --2- 172.21.15.77:0/228981921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3948160a80 0x7f3948069e40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f394806ab70 tx=0x7f3944009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 -- 172.21.15.77:0/228981921 shutdown_connections 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 --2- 172.21.15.77:0/228981921 >> v2:172.21.15.82:3300/0 conn(0x7f394806ea90 0x7f394815d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 --2- 172.21.15.77:0/228981921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3948160a80 0x7f3948069e40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 -- 172.21.15.77:0/228981921 >> 172.21.15.77:0/228981921 conn(0x7f394806af50 msgr2=0x7f394806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 -- 172.21.15.77:0/228981921 shutdown_connections 2023-07-04T18:51:32.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.814+0000 7f394f51c700 1 -- 172.21.15.77:0/228981921 wait complete. 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.815+0000 7f394f51c700 1 Processor -- start 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.815+0000 7f394f51c700 1 -- start start 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.815+0000 7f394f51c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.815+0000 7f394f51c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 0x7f3948148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 0x7f39480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3948148a50 con 0x7f394806a380 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39480ccaa0 con 0x7f394806ea90 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39480ccd20 con 0x7f3948160a80 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44428/0 (socket says 172.21.15.77:44428) 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 -- 172.21.15.77:0/1466370557 learned_addr learned my addr 172.21.15.77:0/1466370557 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394cab7700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 0x7f3948148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 -- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 msgr2=0x7f39480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394dab9700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 0x7f39480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 0x7f39480cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 -- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 msgr2=0x7f3948148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 0x7f3948148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 -- 172.21.15.77:0/1466370557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944009550 con 0x7f394806a380 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394cab7700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 0x7f3948148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394dab9700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 0x7f39480cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:32.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394d2b8700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f393800b530 tx=0x7f393800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938011830 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39380119b0 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938011d60 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39480cd000 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.816+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39480cd5c0 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.818+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394804e210 con 0x7f394806a380 2023-07-04T18:51:32.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.818+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f393801a400 con 0x7f394806a380 2023-07-04T18:51:32.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.818+0000 7f393dffb700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 0x7f392c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.818+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f39380895b0 con 0x7f394806a380 2023-07-04T18:51:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.818+0000 7f394cab7700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 0x7f392c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:32.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.819+0000 7f394cab7700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 0x7f392c06d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3948156380 tx=0x7f3944008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:32.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:32.822+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f39380546f0 con 0x7f394806a380 2023-07-04T18:51:33.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.008+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3948071570 con 0x7f394806a380 2023-07-04T18:51:33.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.008+0000 7f393dffb700 1 -- 172.21.15.77:0/1466370557 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3938057d10 con 0x7f394806a380 2023-07-04T18:51:33.011 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:33.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 msgr2=0x7f392c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 0x7f392c06d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3948156380 tx=0x7f3944008030 comp rx=0 tx=0).stop 2023-07-04T18:51:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 msgr2=0x7f39480cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f393800b530 tx=0x7f393800b8f0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 shutdown_connections 2023-07-04T18:51:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f392c06ace0 0x7f392c06d190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f394806a380 0x7f39480cf230 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f3948160a80 0x7f39480cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 --2- 172.21.15.77:0/1466370557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f394806ea90 0x7f3948148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 >> 172.21.15.77:0/1466370557 conn(0x7f394806af50 msgr2=0x7f394806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 shutdown_connections 2023-07-04T18:51:33.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.010+0000 7f394f51c700 1 -- 172.21.15.77:0/1466370557 wait complete. 2023-07-04T18:51:33.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:33.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.144+0000 7f4c0da77700 1 Processor -- start 2023-07-04T18:51:33.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.144+0000 7f4c0da77700 1 -- start start 2023-07-04T18:51:33.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c0da77700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f4c0806d0b0 0x7f4c0806d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c0da77700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0806d9c0 con 0x7f4c08158e00 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c0da77700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c081592a0 con 0x7f4c080cd250 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c0da77700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c08159eb0 con 0x7f4c0806d0b0 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c06ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4c08158e00 0x7f4c080ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51572/0 (socket says 172.21.15.77:51572) 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c06ffd700 1 -- 172.21.15.77:0/2747781677 learned_addr learned my addr 172.21.15.77:0/2747781677 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:33.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c077fe700 1 --2- 172.21.15.77:0/2747781677 >> v2:172.21.15.82:3300/0 conn(0x7f4c0806d0b0 0x7f4c0806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371034175 0 0) 0x7f4c0806d9c0 con 0x7f4c08158e00 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.145+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf00035b0 con 0x7f4c08158e00 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154224614 0 0) 0x7f4bf00035b0 con 0x7f4c08158e00 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 >> v2:172.21.15.82:3300/0 conn(0x7f4c0806d0b0 msgr2=0x7f4c0806d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 --2- 172.21.15.77:0/2747781677 >> v2:172.21.15.82:3300/0 conn(0x7f4c0806d0b0 0x7f4c0806d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 >> v1:172.21.15.82:6789/0 conn(0x7f4c080cd250 legacy=0x7f4c080cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c08159580 con 0x7f4c08158e00 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f4bf8001670 con 0x7f4c08158e00 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 >> v1:172.21.15.77:6789/0 conn(0x7f4c08158e00 legacy=0x7f4c080ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bf0003a00 0x7f4bf0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4bf0004850 0x7f4bf0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4bf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c08159eb0 con 0x7f4bf0005200 2023-07-04T18:51:33.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf000c110 con 0x7f4bf0003a00 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf000c2d0 con 0x7f4bf0004850 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4bf000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c06ffd700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4bf0004850 0x7f4bf0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c077fe700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bf0003a00 0x7f4bf0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 -- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4bf0004850 msgr2=0x7f4bf0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4bf0004850 0x7f4bf0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 -- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bf0003a00 msgr2=0x7f4bf0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bf0003a00 0x7f4bf0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 -- 172.21.15.77:0/2747781677 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf00035b0 con 0x7f4bf0005200 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 -- 172.21.15.77:0/2747781677 reap_dead start 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c07fff700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4bf000ba10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c190 tx=0x7f4bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932dae0aeb4652b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00001db0 con 0x7f4bf0005200 2023-07-04T18:51:33.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0000ccf0 con 0x7f4bf0005200 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.146+0000 7f4c067fc700 1 -- 172.21.15.77:0/2747781677 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00003c40 con 0x7f4bf0005200 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 -- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 msgr2=0x7f4bf000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4bf000ba10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c190 tx=0x7f4bf0005880 comp rx=0 tx=0).stop 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 -- 172.21.15.77:0/2747781677 shutdown_connections 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 --2- 172.21.15.77:0/2747781677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4bf000ba10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 -- 172.21.15.77:0/2747781677 >> 172.21.15.77:0/2747781677 conn(0x7f4c080cfcb0 msgr2=0x7f4c080cbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 -- 172.21.15.77:0/2747781677 shutdown_connections 2023-07-04T18:51:33.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.147+0000 7f4c0da77700 1 -- 172.21.15.77:0/2747781677 wait complete. 2023-07-04T18:51:33.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.148+0000 7f4c0da77700 1 Processor -- start 2023-07-04T18:51:33.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.148+0000 7f4c0da77700 1 -- start start 2023-07-04T18:51:33.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4c081e2460 0x7f4c08149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 0x7f4c080c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c080b47a0 con 0x7f4bf0005200 2023-07-04T18:51:33.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c080b4fb0 con 0x7f4c080b20f0 2023-07-04T18:51:33.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c0da77700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c080b4020 con 0x7f4c081e2460 2023-07-04T18:51:33.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44446/0 (socket says 172.21.15.77:44446) 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 -- 172.21.15.77:0/224149533 learned_addr learned my addr 172.21.15.77:0/224149533 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c07fff700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 0x7f4c080c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c06ffd700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4c081e2460 0x7f4c08149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 -- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4c081e2460 msgr2=0x7f4c08149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4c081e2460 0x7f4c08149360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 -- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 msgr2=0x7f4c080c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 0x7f4c080c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c077fe700 1 -- 172.21.15.77:0/224149533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c080b49f0 con 0x7f4bf0005200 2023-07-04T18:51:33.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.149+0000 7f4c07fff700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 0x7f4c080c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:33.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4c077fe700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc011a00 tx=0x7f4bfc011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00fc40 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc00fda0 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c080b4cd0 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc01c6a0 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.150+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c081498a0 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.151+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0804e210 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.151+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4bfc01c800 con 0x7f4bf0005200 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.151+0000 7f4beffff700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 0x7f4bf406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.152+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4bfc017070 con 0x7f4bf0005200 2023-07-04T18:51:33.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.152+0000 7f4c06ffd700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 0x7f4bf406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.152+0000 7f4c06ffd700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 0x7f4bf406d190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8001050 tx=0x7f4bf8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.156+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4bfc0550c0 con 0x7f4bf0005200 2023-07-04T18:51:33.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.343+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c080ce110 con 0x7f4bf0005200 2023-07-04T18:51:33.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.344+0000 7f4beffff700 1 -- 172.21.15.77:0/224149533 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4bfc0586e0 con 0x7f4bf0005200 2023-07-04T18:51:33.346 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:33.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 msgr2=0x7f4bf406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 0x7f4bf406d190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8001050 tx=0x7f4bf8009040 comp rx=0 tx=0).stop 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 msgr2=0x7f4c080b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc011a00 tx=0x7f4bfc011dc0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 shutdown_connections 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f4bf406ace0 0x7f4bf406d190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4bf0005200 0x7f4c080b3ae0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c080b20f0 0x7f4c080c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 --2- 172.21.15.77:0/224149533 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4c081e2460 0x7f4c08149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 >> 172.21.15.77:0/224149533 conn(0x7f4c080cfcb0 msgr2=0x7f4c080ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 shutdown_connections 2023-07-04T18:51:33.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.346+0000 7f4c0da77700 1 -- 172.21.15.77:0/224149533 wait complete. 2023-07-04T18:51:33.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:33.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.479+0000 7f1c56af6700 1 Processor -- start 2023-07-04T18:51:33.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 -- start start 2023-07-04T18:51:33.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c500aded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c500708a0 0x7f1c50070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500ca8a0 con 0x7f1c500adb00 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500cce60 con 0x7f1c500708a0 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c56af6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50071260 con 0x7f1c5006d5d0 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.480+0000 7f1c55093700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1c500708a0 0x7f1c50070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c54892700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1c500adb00 0x7f1c500ca190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51592/0 (socket says 172.21.15.77:51592) 2023-07-04T18:51:33.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c54892700 1 -- 172.21.15.77:0/2709993809 learned_addr learned my addr 172.21.15.77:0/2709993809 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:33.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c4ffff700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c500aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c55093700 1 -- 172.21.15.77:0/2709993809 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 msgr2=0x7f1c500aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c55093700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c500aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c55093700 1 -- 172.21.15.77:0/2709993809 >> v1:172.21.15.77:6789/0 conn(0x7f1c500adb00 legacy=0x7f1c500ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c4f7fe700 1 -- 172.21.15.77:0/2709993809 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258112173 0 0) 0x7f1c500ca8a0 con 0x7f1c500adb00 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c55093700 1 -- 172.21.15.77:0/2709993809 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c500cb3b0 con 0x7f1c500708a0 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c55093700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c50070d20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009d90 tx=0x7f1c4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6137adc39805f7a server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c4f7fe700 1 -- 172.21.15.77:0/2709993809 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4800e070 con 0x7f1c500708a0 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c4f7fe700 1 -- 172.21.15.77:0/2709993809 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800bbd0 con 0x7f1c500708a0 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.481+0000 7f1c4f7fe700 1 -- 172.21.15.77:0/2709993809 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48012430 con 0x7f1c500708a0 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 -- 172.21.15.77:0/2709993809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 msgr2=0x7f1c50070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c50070d20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009d90 tx=0x7f1c4800b330 comp rx=0 tx=0).stop 2023-07-04T18:51:33.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 -- 172.21.15.77:0/2709993809 shutdown_connections 2023-07-04T18:51:33.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c50070d20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 --2- 172.21.15.77:0/2709993809 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c500aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 -- 172.21.15.77:0/2709993809 >> 172.21.15.77:0/2709993809 conn(0x7f1c500cf0d0 msgr2=0x7f1c500cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 -- 172.21.15.77:0/2709993809 shutdown_connections 2023-07-04T18:51:33.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.482+0000 7f1c56af6700 1 -- 172.21.15.77:0/2709993809 wait complete. 2023-07-04T18:51:33.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.483+0000 7f1c56af6700 1 Processor -- start 2023-07-04T18:51:33.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.483+0000 7f1c56af6700 1 -- start start 2023-07-04T18:51:33.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c50146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 0x7f1c500ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500c71f0 con 0x7f1c500708a0 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5014a5e0 con 0x7f1c500adb00 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500c7770 con 0x7f1c5006d5d0 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44462/0 (socket says 172.21.15.77:44462) 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 -- 172.21.15.77:0/4046571724 learned_addr learned my addr 172.21.15.77:0/4046571724 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:33.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c54892700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c50146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c55093700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 0x7f1c500ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 -- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 msgr2=0x7f1c50146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c50146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 -- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 msgr2=0x7f1c500ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 0x7f1c500ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 -- 172.21.15.77:0/4046571724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009a70 con 0x7f1c500708a0 2023-07-04T18:51:33.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c55093700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 0x7f1c500ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4ffff700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00fad0 tx=0x7f1c3c00fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c007b40 con 0x7f1c500708a0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c007ca0 con 0x7f1c500708a0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c501f1920 con 0x7f1c500708a0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c01a6a0 con 0x7f1c500708a0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.484+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c500c7400 con 0x7f1c500708a0 2023-07-04T18:51:33.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.485+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5004e210 con 0x7f1c500708a0 2023-07-04T18:51:33.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.486+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1c3c011450 con 0x7f1c500708a0 2023-07-04T18:51:33.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.486+0000 7f1c4d7fa700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 0x7f1c3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.486+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1c3c08bb20 con 0x7f1c500708a0 2023-07-04T18:51:33.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.487+0000 7f1c54892700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 0x7f1c3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.487+0000 7f1c54892700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 0x7f1c3406d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1c44000f60 tx=0x7f1c44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.491+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1c3c056bd0 con 0x7f1c500708a0 2023-07-04T18:51:33.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.677+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1c500c9f70 con 0x7f1c500708a0 2023-07-04T18:51:33.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.677+0000 7f1c4d7fa700 1 -- 172.21.15.77:0/4046571724 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1c3c015020 con 0x7f1c500708a0 2023-07-04T18:51:33.679 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:33.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 msgr2=0x7f1c3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 0x7f1c3406d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1c44000f60 tx=0x7f1c44009040 comp rx=0 tx=0).stop 2023-07-04T18:51:33.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 msgr2=0x7f1c501f0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00fad0 tx=0x7f1c3c00fe90 comp rx=0 tx=0).stop 2023-07-04T18:51:33.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 shutdown_connections 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f1c3406ace0 0x7f1c3406d190 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c500adb00 0x7f1c500ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1c5006d5d0 0x7f1c50146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 --2- 172.21.15.77:0/4046571724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c500708a0 0x7f1c501f0e80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 >> 172.21.15.77:0/4046571724 conn(0x7f1c500cf0d0 msgr2=0x7f1c501584f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 shutdown_connections 2023-07-04T18:51:33.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.679+0000 7f1c56af6700 1 -- 172.21.15.77:0/4046571724 wait complete. 2023-07-04T18:51:33.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:51:33.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 Processor -- start 2023-07-04T18:51:33.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 -- start start 2023-07-04T18:51:33.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 0x7ff6cc0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6cc0cf100 con 0x7ff6cc0cebf0 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6cc15df60 con 0x7ff6cc15a430 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.811+0000 7ff6d3755700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6cc15e240 con 0x7ff6cc15ea40 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44468/0 (socket says 172.21.15.77:44468) 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 -- 172.21.15.77:0/2740949263 learned_addr learned my addr 172.21.15.77:0/2740949263 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:33.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d1cf2700 1 --2- 172.21.15.77:0/2740949263 >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 0x7ff6cc0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 -- 172.21.15.77:0/2740949263 >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 msgr2=0x7ff6cc0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 --2- 172.21.15.77:0/2740949263 >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 0x7ff6cc0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 -- 172.21.15.77:0/2740949263 >> v1:172.21.15.82:6789/0 conn(0x7ff6cc0cebf0 legacy=0x7ff6cc0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 -- 172.21.15.77:0/2740949263 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc15e520 con 0x7ff6cc15a430 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d1cf2700 1 --2- 172.21.15.77:0/2740949263 >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 0x7ff6cc0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6d14f1700 1 --2- 172.21.15.77:0/2740949263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0b04f0 tx=0x7ff6bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6224c61cb00386a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6c3fff700 1 -- 172.21.15.77:0/2740949263 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc004230 con 0x7ff6cc15a430 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6c3fff700 1 -- 172.21.15.77:0/2740949263 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc004390 con 0x7ff6cc15a430 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.812+0000 7ff6c3fff700 1 -- 172.21.15.77:0/2740949263 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0036a0 con 0x7ff6cc15a430 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 -- 172.21.15.77:0/2740949263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15a430 msgr2=0x7ff6cc0ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 --2- 172.21.15.77:0/2740949263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0b04f0 tx=0x7ff6bc009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 -- 172.21.15.77:0/2740949263 shutdown_connections 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 --2- 172.21.15.77:0/2740949263 >> v2:172.21.15.82:3300/0 conn(0x7ff6cc15ea40 0x7ff6cc0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 --2- 172.21.15.77:0/2740949263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15a430 0x7ff6cc0ce6b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 -- 172.21.15.77:0/2740949263 >> 172.21.15.77:0/2740949263 conn(0x7ff6cc06d7c0 msgr2=0x7ff6cc06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:33.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 -- 172.21.15.77:0/2740949263 shutdown_connections 2023-07-04T18:51:33.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.813+0000 7ff6d3755700 1 -- 172.21.15.77:0/2740949263 wait complete. 2023-07-04T18:51:33.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.814+0000 7ff6d3755700 1 Processor -- start 2023-07-04T18:51:33.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.814+0000 7ff6d3755700 1 -- start start 2023-07-04T18:51:33.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6cc0cebf0 0x7ff6cc0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 0x7ff6cc1e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc1e4620 con 0x7ff6cc15ea40 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0b53e0 con 0x7ff6cc0cebf0 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0b47d0 con 0x7ff6cc1e3c40 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44476/0 (socket says 172.21.15.77:44476) 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 -- 172.21.15.77:0/4223955560 learned_addr learned my addr 172.21.15.77:0/4223955560 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d14f1700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6cc0cebf0 0x7ff6cc0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d1cf2700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 0x7ff6cc1e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 -- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 msgr2=0x7ff6cc1e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 0x7ff6cc1e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 -- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6cc0cebf0 msgr2=0x7ff6cc0b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6cc0cebf0 0x7ff6cc0b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:33.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 -- 172.21.15.77:0/4223955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc009550 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d1cf2700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 0x7ff6cc1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d0cf0700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c4a0 tx=0x7ff6c800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80054e0 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8003680 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc0b4ab0 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.815+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003a30 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.816+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc0b5070 con 0x7ff6cc15ea40 2023-07-04T18:51:33.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.816+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc04e210 con 0x7ff6cc15ea40 2023-07-04T18:51:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.817+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff6c8003b90 con 0x7ff6cc15ea40 2023-07-04T18:51:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.817+0000 7ff6c1ffb700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 0x7ff6b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.817+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff6c80889a0 con 0x7ff6cc15ea40 2023-07-04T18:51:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.817+0000 7ff6d14f1700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 0x7ff6b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:33.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.818+0000 7ff6d14f1700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 0x7ff6b006d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009130 tx=0x7ff6bc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:33.825 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:33.822+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff6c80539c0 con 0x7ff6cc15ea40 2023-07-04T18:51:34.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.050+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6cc0b5520 con 0x7ff6cc15ea40 2023-07-04T18:51:34.052 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.051+0000 7ff6c1ffb700 1 -- 172.21.15.77:0/4223955560 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7ff6c8017070 con 0x7ff6cc15ea40 2023-07-04T18:51:34.053 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:34.053 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:51:34.054 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 msgr2=0x7ff6b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 0x7ff6b006d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009130 tx=0x7ff6bc008030 comp rx=0 tx=0).stop 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 msgr2=0x7ff6cc153b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c4a0 tx=0x7ff6c800c7b0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 shutdown_connections 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff6b006ace0 0x7ff6b006d190 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff6cc1e3c40 0x7ff6cc1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.055 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6cc0cebf0 0x7ff6cc0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 --2- 172.21.15.77:0/4223955560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6cc15ea40 0x7ff6cc153b40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 >> 172.21.15.77:0/4223955560 conn(0x7ff6cc06d7c0 msgr2=0x7ff6cc154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 shutdown_connections 2023-07-04T18:51:34.056 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.053+0000 7ff6d3755700 1 -- 172.21.15.77:0/4223955560 wait complete. 2023-07-04T18:51:34.056 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:51:34.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 Processor -- start 2023-07-04T18:51:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 -- start start 2023-07-04T18:51:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbbf0158cc0 0x7fbbf006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf00cb230 0x7fbbf00cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf00cbbf0 con 0x7fbbf006cbe0 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf00cdd70 con 0x7fbbf0158cc0 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.186+0000 7fbbf542d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00ce050 con 0x7fbbf00cb230 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbee7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbbf006cbe0 0x7fbbf00710e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51620/0 (socket says 172.21.15.77:51620) 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbee7fc700 1 -- 172.21.15.77:0/3359189954 learned_addr learned my addr 172.21.15.77:0/3359189954 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 --2- 172.21.15.77:0/3359189954 >> v2:172.21.15.82:3300/0 conn(0x7fbbf0158cc0 0x7fbbf006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbef7fe700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf00cb230 0x7fbbf00cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbedffb700 1 -- 172.21.15.77:0/3359189954 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691602228 0 0) 0x7fbbf00cbbf0 con 0x7fbbf006cbe0 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbedffb700 1 -- 172.21.15.77:0/3359189954 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd80035b0 con 0x7fbbf006cbe0 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 -- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf00cb230 msgr2=0x7fbbf00cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf00cb230 0x7fbbf00cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 -- 172.21.15.77:0/3359189954 >> v1:172.21.15.77:6789/0 conn(0x7fbbf006cbe0 legacy=0x7fbbf00710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 -- 172.21.15.77:0/3359189954 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf00ce330 con 0x7fbbf0158cc0 2023-07-04T18:51:34.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbeeffd700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf0158cc0 0x7fbbf006f550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009870 tx=0x7fbbe0009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19e622544050742 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbedffb700 1 -- 172.21.15.77:0/3359189954 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0011830 con 0x7fbbf0158cc0 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbedffb700 1 -- 172.21.15.77:0/3359189954 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe0011990 con 0x7fbbf0158cc0 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.187+0000 7fbbedffb700 1 -- 172.21.15.77:0/3359189954 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0011d00 con 0x7fbbf0158cc0 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 -- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf0158cc0 msgr2=0x7fbbf006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf0158cc0 0x7fbbf006f550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009870 tx=0x7fbbe0009b80 comp rx=0 tx=0).stop 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 -- 172.21.15.77:0/3359189954 shutdown_connections 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf00cb230 0x7fbbf00cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 --2- 172.21.15.77:0/3359189954 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf0158cc0 0x7fbbf006f550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 -- 172.21.15.77:0/3359189954 >> 172.21.15.77:0/3359189954 conn(0x7fbbf006aad0 msgr2=0x7fbbf00b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 -- 172.21.15.77:0/3359189954 shutdown_connections 2023-07-04T18:51:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.188+0000 7fbbf542d700 1 -- 172.21.15.77:0/3359189954 wait complete. 2023-07-04T18:51:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.189+0000 7fbbf542d700 1 Processor -- start 2023-07-04T18:51:34.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.189+0000 7fbbf542d700 1 -- start start 2023-07-04T18:51:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf006cbe0 0x7fbbf00af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 0x7fbbf00b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00afbd0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00b3660 con 0x7fbbf00cb230 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbf542d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00b3940 con 0x7fbbf006cbe0 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbee7fc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 0x7fbbf00b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44492/0 (socket says 172.21.15.77:44492) 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbeeffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf006cbe0 0x7fbbf00af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 -- 172.21.15.77:0/514693589 learned_addr learned my addr 172.21.15.77:0/514693589 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 -- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf006cbe0 msgr2=0x7fbbf00af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf006cbe0 0x7fbbf00af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 -- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 msgr2=0x7fbbf00b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 0x7fbbf00b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbef7fe700 1 -- 172.21.15.77:0/514693589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00094a0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.190+0000 7fbbee7fc700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 0x7fbbf00b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbef7fe700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800efc0 tx=0x7fbbe800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8009ea0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe800f040 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf00b3c20 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe800ced0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.191+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf00b41e0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.201 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.192+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf00b4400 con 0x7fbbf0158cc0 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.192+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbbe8014480 con 0x7fbbf0158cc0 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.193+0000 7fbbd77fe700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 0x7fbbdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.193+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbbe8089cf0 con 0x7fbbf0158cc0 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.193+0000 7fbbeeffd700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 0x7fbbdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.193+0000 7fbbeeffd700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 0x7fbbdc06d190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00090b0 tx=0x7fbbe001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.197+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbbe808e070 con 0x7fbbf0158cc0 2023-07-04T18:51:34.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.385+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf004e210 con 0x7fbbf0158cc0 2023-07-04T18:51:34.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.386+0000 7fbbd77fe700 1 -- 172.21.15.77:0/514693589 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7fbbe8058830 con 0x7fbbf0158cc0 2023-07-04T18:51:34.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:34.389 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:34.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 msgr2=0x7fbbdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 0x7fbbdc06d190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00090b0 tx=0x7fbbe001c040 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 msgr2=0x7fbbf00ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800efc0 tx=0x7fbbe800c3a0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 shutdown_connections 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbbdc06ace0 0x7fbbdc06d190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbf0158cc0 0x7fbbf00ac8b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbbf006cbe0 0x7fbbf00af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 --2- 172.21.15.77:0/514693589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbf00cb230 0x7fbbf00b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.391 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 >> 172.21.15.77:0/514693589 conn(0x7fbbf006aad0 msgr2=0x7fbbf00cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 shutdown_connections 2023-07-04T18:51:34.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.388+0000 7fbbf542d700 1 -- 172.21.15.77:0/514693589 wait complete. 2023-07-04T18:51:34.402 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:34.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.523+0000 7f34e8959700 1 Processor -- start 2023-07-04T18:51:34.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.523+0000 7f34e8959700 1 -- start start 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e8959700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f34e4069e40 0x7f34e406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e8959700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e41582f0 con 0x7f34e4157f20 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e8959700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e40d0780 con 0x7f34e40d0300 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e8959700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e406a820 con 0x7f34e4069e40 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e259c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f34e4157f20 0x7f34e415a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51634/0 (socket says 172.21.15.77:51634) 2023-07-04T18:51:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e259c700 1 -- 172.21.15.77:0/178768317 learned_addr learned my addr 172.21.15.77:0/178768317 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/178768317 >> v2:172.21.15.82:3300/0 conn(0x7f34e4069e40 0x7f34e406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991537244 0 0) 0x7f34e41582f0 con 0x7f34e4157f20 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.524+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc0035b0 con 0x7f34e4157f20 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682242819 0 0) 0x7f34cc0035b0 con 0x7f34e4157f20 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 >> v2:172.21.15.82:3300/0 conn(0x7f34e4069e40 msgr2=0x7f34e406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 --2- 172.21.15.77:0/178768317 >> v2:172.21.15.82:3300/0 conn(0x7f34e4069e40 0x7f34e406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 >> v1:172.21.15.82:6789/0 conn(0x7f34e40d0300 legacy=0x7f34e415e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e415b0b0 con 0x7f34e4157f20 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f34d8001670 con 0x7f34e4157f20 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/178768317 >> v2:172.21.15.82:3300/0 conn(0x7f34e4069e40 0x7f34e406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 >> v1:172.21.15.77:6789/0 conn(0x7f34e4157f20 legacy=0x7f34e415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 0x7f34cc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34cc004850 0x7f34cc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc005200 0x7f34cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc00c0e0 con 0x7f34cc003a00 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc00c250 con 0x7f34cc004850 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc00c410 con 0x7f34cc005200 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 0x7f34cc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e1d9b700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34cc004850 0x7f34cc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc005200 0x7f34cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 -- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc005200 msgr2=0x7f34cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc005200 0x7f34cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 -- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34cc004850 msgr2=0x7f34cc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34cc004850 0x7f34cc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 -- 172.21.15.77:0/178768317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0035b0 con 0x7f34cc003a00 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e2d9d700 1 -- 172.21.15.77:0/178768317 reap_dead start 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e259c700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 0x7f34cc003e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c240 tx=0x7f34cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa71a1690a753661 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8007c10 con 0x7f34cc003a00 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d800cd80 con 0x7f34cc003a00 2023-07-04T18:51:34.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.525+0000 7f34e159a700 1 -- 172.21.15.77:0/178768317 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8005ba0 con 0x7f34cc003a00 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 -- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 msgr2=0x7f34cc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 0x7f34cc003e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c240 tx=0x7f34cc005880 comp rx=0 tx=0).stop 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 -- 172.21.15.77:0/178768317 shutdown_connections 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 --2- 172.21.15.77:0/178768317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34cc003a00 0x7f34cc003e10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 -- 172.21.15.77:0/178768317 >> 172.21.15.77:0/178768317 conn(0x7f34e406af50 msgr2=0x7f34e406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 -- 172.21.15.77:0/178768317 shutdown_connections 2023-07-04T18:51:34.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.526+0000 7f34e8959700 1 -- 172.21.15.77:0/178768317 wait complete. 2023-07-04T18:51:34.535 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.527+0000 7f34e8959700 1 Processor -- start 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.527+0000 7f34e8959700 1 -- start start 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 0x7f34e4159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34e414ebe0 0x7f34e4154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4158ea0 con 0x7f34e41554d0 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4159180 con 0x7f34e414ebe0 2023-07-04T18:51:34.536 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e8959700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e4159460 con 0x7f34cc003a00 2023-07-04T18:51:34.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44512/0 (socket says 172.21.15.77:44512) 2023-07-04T18:51:34.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 -- 172.21.15.77:0/496964028 learned_addr learned my addr 172.21.15.77:0/496964028 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:34.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e1d9b700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34e414ebe0 0x7f34e4154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.537 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e259c700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 0x7f34e4159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 -- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 msgr2=0x7f34e4159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 0x7f34e4159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 -- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34e414ebe0 msgr2=0x7f34e4154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34e414ebe0 0x7f34e4154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e2d9d700 1 -- 172.21.15.77:0/496964028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4159740 con 0x7f34e41554d0 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.528+0000 7f34e259c700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 0x7f34e4159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:34.539 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34e2d9d700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34e40af5c0 tx=0x7f34dc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc00e070 con 0x7f34e41554d0 2023-07-04T18:51:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc003c30 con 0x7f34e41554d0 2023-07-04T18:51:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e4159a20 con 0x7f34e41554d0 2023-07-04T18:51:34.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc003dc0 con 0x7f34e41554d0 2023-07-04T18:51:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.529+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e4154960 con 0x7f34e41554d0 2023-07-04T18:51:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.530+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e404e210 con 0x7f34e41554d0 2023-07-04T18:51:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.530+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f34dc011a60 con 0x7f34e41554d0 2023-07-04T18:51:34.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.530+0000 7f34d2ffd700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 0x7f34c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.531+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f34dc089950 con 0x7f34e41554d0 2023-07-04T18:51:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.531+0000 7f34e259c700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 0x7f34c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.531+0000 7f34e259c700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 0x7f34c406d190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.535+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f34dc054a00 con 0x7f34e41554d0 2023-07-04T18:51:34.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.722+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34e4071080 con 0x7f34e41554d0 2023-07-04T18:51:34.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.722+0000 7f34d2ffd700 1 -- 172.21.15.77:0/496964028 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34dc058020 con 0x7f34e41554d0 2023-07-04T18:51:34.725 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:34.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 msgr2=0x7f34c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 0x7f34c406d190 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f34d8000c00 tx=0x7f34d8007000 comp rx=0 tx=0).stop 2023-07-04T18:51:34.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 msgr2=0x7f34e406cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f34e40af5c0 tx=0x7f34dc009f90 comp rx=0 tx=0).stop 2023-07-04T18:51:34.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 shutdown_connections 2023-07-04T18:51:34.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34c406ace0 0x7f34c406d190 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34e41554d0 0x7f34e406cc30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34cc003a00 0x7f34e4159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 --2- 172.21.15.77:0/496964028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34e414ebe0 0x7f34e4154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 >> 172.21.15.77:0/496964028 conn(0x7f34e406af50 msgr2=0x7f34e4150700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 shutdown_connections 2023-07-04T18:51:34.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.724+0000 7f34e8959700 1 -- 172.21.15.77:0/496964028 wait complete. 2023-07-04T18:51:34.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:34.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 Processor -- start 2023-07-04T18:51:34.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 -- start start 2023-07-04T18:51:34.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f5f140b0360 0x7f5f14160b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f140cd700 0x7f5f140cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f140cefd0 con 0x7f5f140b2800 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140cf7d0 con 0x7f5f140cd700 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.864+0000 7f5f19506700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1415caf0 con 0x7f5f140b0360 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f140cd700 0x7f5f140cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f127fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5f140b2800 0x7f5f140ce8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51654/0 (socket says 172.21.15.77:51654) 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f127fc700 1 -- 172.21.15.77:0/1366081595 learned_addr learned my addr 172.21.15.77:0/1366081595 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f12ffd700 1 --2- 172.21.15.77:0/1366081595 >> v2:172.21.15.82:3300/0 conn(0x7f5f140b0360 0x7f5f14160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 -- 172.21.15.77:0/1366081595 >> v2:172.21.15.82:3300/0 conn(0x7f5f140b0360 msgr2=0x7f5f14160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 --2- 172.21.15.77:0/1366081595 >> v2:172.21.15.82:3300/0 conn(0x7f5f140b0360 0x7f5f14160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 -- 172.21.15.77:0/1366081595 >> v1:172.21.15.77:6789/0 conn(0x7f5f140b2800 legacy=0x7f5f140ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f11ffb700 1 -- 172.21.15.77:0/1366081595 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025454418 0 0) 0x7f5f08001110 con 0x7f5f140b2800 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 -- 172.21.15.77:0/1366081595 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f141606d0 con 0x7f5f140cd700 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f137fe700 1 --2- 172.21.15.77:0/1366081595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140cd700 0x7f5f140cdb80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f140cd6d0 tx=0x7f5f0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc729329ba7028b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f11ffb700 1 -- 172.21.15.77:0/1366081595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c00e070 con 0x7f5f140cd700 2023-07-04T18:51:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f11ffb700 1 -- 172.21.15.77:0/1366081595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c00bbd0 con 0x7f5f140cd700 2023-07-04T18:51:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.865+0000 7f5f11ffb700 1 -- 172.21.15.77:0/1366081595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c012430 con 0x7f5f140cd700 2023-07-04T18:51:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 -- 172.21.15.77:0/1366081595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140cd700 msgr2=0x7f5f140cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 --2- 172.21.15.77:0/1366081595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140cd700 0x7f5f140cdb80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f140cd6d0 tx=0x7f5f0c00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 -- 172.21.15.77:0/1366081595 shutdown_connections 2023-07-04T18:51:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 --2- 172.21.15.77:0/1366081595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140cd700 0x7f5f140cdb80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 --2- 172.21.15.77:0/1366081595 >> v2:172.21.15.82:3300/0 conn(0x7f5f140b0360 0x7f5f14160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 -- 172.21.15.77:0/1366081595 >> 172.21.15.77:0/1366081595 conn(0x7f5f1406d900 msgr2=0x7f5f1406dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 -- 172.21.15.77:0/1366081595 shutdown_connections 2023-07-04T18:51:34.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.866+0000 7f5f19506700 1 -- 172.21.15.77:0/1366081595 wait complete. 2023-07-04T18:51:34.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.867+0000 7f5f19506700 1 Processor -- start 2023-07-04T18:51:34.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.867+0000 7f5f19506700 1 -- start start 2023-07-04T18:51:34.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 0x7f5f14153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f140cd700 0x7f5f140b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1415b020 con 0x7f5f140b0360 2023-07-04T18:51:34.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1415ad40 con 0x7f5f140cd700 2023-07-04T18:51:34.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140c4750 con 0x7f5f140b2800 2023-07-04T18:51:34.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44520/0 (socket says 172.21.15.77:44520) 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 -- 172.21.15.77:0/3007765726 learned_addr learned my addr 172.21.15.77:0/3007765726 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f137fe700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f140cd700 0x7f5f140b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f127fc700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 0x7f5f14153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 -- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 msgr2=0x7f5f14153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 0x7f5f14153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 -- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f140cd700 msgr2=0x7f5f140b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f140cd700 0x7f5f140b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 -- 172.21.15.77:0/3007765726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c009a70 con 0x7f5f140b0360 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f127fc700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 0x7f5f14153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f12ffd700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400c7f0 tx=0x7f5f0400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04004220 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f04004380 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f141f6a80 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f040036a0 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.868+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1415a9d0 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.869+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1404e210 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.870+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5f04003800 con 0x7f5f140b0360 2023-07-04T18:51:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.870+0000 7f5efb7fe700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 0x7f5f0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.870+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5f04089060 con 0x7f5f140b0360 2023-07-04T18:51:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.870+0000 7f5f127fc700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 0x7f5f0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.871+0000 7f5f127fc700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 0x7f5f0006d190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000f60 tx=0x7f5f08007ae0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:34.874+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5f04057bb0 con 0x7f5f140b0360 2023-07-04T18:51:35.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.064+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5f140b3110 con 0x7f5f140b0360 2023-07-04T18:51:35.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.065+0000 7f5efb7fe700 1 -- 172.21.15.77:0/3007765726 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5f04057740 con 0x7f5f140b0360 2023-07-04T18:51:35.066 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:35.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 msgr2=0x7f5f0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 0x7f5f0006d190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000f60 tx=0x7f5f08007ae0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 msgr2=0x7f5f141f6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400c7f0 tx=0x7f5f0400cb00 comp rx=0 tx=0).stop 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 shutdown_connections 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f0006ace0 0x7f5f0006d190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f140b0360 0x7f5f141f6540 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f140cd700 0x7f5f140b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 --2- 172.21.15.77:0/3007765726 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f140b2800 0x7f5f14153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 >> 172.21.15.77:0/3007765726 conn(0x7f5f1406d900 msgr2=0x7f5f140d1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 shutdown_connections 2023-07-04T18:51:35.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.067+0000 7f5f19506700 1 -- 172.21.15.77:0/3007765726 wait complete. 2023-07-04T18:51:35.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:35.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.205+0000 7fc8429cc700 1 Processor -- start 2023-07-04T18:51:35.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.205+0000 7fc8429cc700 1 -- start start 2023-07-04T18:51:35.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0b0360 0x7fc83c160b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 0x7fc83c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0cd700 0x7fc83c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c0cf7d0 con 0x7fc83c0cd700 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83c15caf0 con 0x7fc83c0b2800 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc8429cc700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c0ce0e0 con 0x7fc83c0b0360 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0cd700 0x7fc83c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc83c0cd700 0x7fc83c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44532/0 (socket says 172.21.15.77:44532) 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 -- 172.21.15.77:0/3063341968 learned_addr learned my addr 172.21.15.77:0/3063341968 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc83b7fe700 1 --2- 172.21.15.77:0/3063341968 >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 0x7fc83c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc83bfff700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0b0360 0x7fc83c160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 -- 172.21.15.77:0/3063341968 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0b0360 msgr2=0x7fc83c160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0b0360 0x7fc83c160b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 -- 172.21.15.77:0/3063341968 >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 msgr2=0x7fc83c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 --2- 172.21.15.77:0/3063341968 >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 0x7fc83c0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 -- 172.21.15.77:0/3063341968 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c1606d0 con 0x7fc83c0cd700 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc83b7fe700 1 --2- 172.21.15.77:0/3063341968 >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 0x7fc83c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc840f69700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0cd700 0x7fc83c0cdba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0d1dc0 tx=0x7fc83400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f2c6c29de4bdfa server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.206+0000 7fc83affd700 1 -- 172.21.15.77:0/3063341968 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400e070 con 0x7fc83c0cd700 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc83affd700 1 -- 172.21.15.77:0/3063341968 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83400b9c0 con 0x7fc83c0cd700 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc83affd700 1 -- 172.21.15.77:0/3063341968 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400bd70 con 0x7fc83c0cd700 2023-07-04T18:51:35.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 -- 172.21.15.77:0/3063341968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0cd700 msgr2=0x7fc83c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0cd700 0x7fc83c0cdba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc83c0d1dc0 tx=0x7fc83400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 -- 172.21.15.77:0/3063341968 shutdown_connections 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0cd700 0x7fc83c0cdba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 --2- 172.21.15.77:0/3063341968 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0b0360 0x7fc83c160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 --2- 172.21.15.77:0/3063341968 >> v2:172.21.15.82:3300/0 conn(0x7fc83c0b2800 0x7fc83c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 -- 172.21.15.77:0/3063341968 >> 172.21.15.77:0/3063341968 conn(0x7fc83c06d900 msgr2=0x7fc83c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 -- 172.21.15.77:0/3063341968 shutdown_connections 2023-07-04T18:51:35.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.207+0000 7fc8429cc700 1 -- 172.21.15.77:0/3063341968 wait complete. 2023-07-04T18:51:35.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.209+0000 7fc8429cc700 1 Processor -- start 2023-07-04T18:51:35.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.209+0000 7fc8429cc700 1 -- start start 2023-07-04T18:51:35.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc83c0b0360 0x7fc83c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0cd700 0x7fc83c0b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15ec20 con 0x7fc83c0b2800 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15ef00 con 0x7fc83c0b0360 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c15f1e0 con 0x7fc83c0cd700 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44536/0 (socket says 172.21.15.77:44536) 2023-07-04T18:51:35.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 -- 172.21.15.77:0/837792078 learned_addr learned my addr 172.21.15.77:0/837792078 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83bfff700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc83c0b0360 0x7fc83c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc840f69700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0cd700 0x7fc83c0b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 -- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0cd700 msgr2=0x7fc83c0b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0cd700 0x7fc83c0b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 -- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc83c0b0360 msgr2=0x7fc83c0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc83c0b0360 0x7fc83c0b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 -- 172.21.15.77:0/837792078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834009b30 con 0x7fc83c0b2800 2023-07-04T18:51:35.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc83b7fe700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c750 tx=0x7fc82c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c011800 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c011960 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c15f4c0 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c011cd0 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.210+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c0b4a70 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.211+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c04e210 con 0x7fc83c0b2800 2023-07-04T18:51:35.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.212+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc82c021350 con 0x7fc83c0b2800 2023-07-04T18:51:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.212+0000 7fc838ff9700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 0x7fc82806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.212+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc82c089460 con 0x7fc83c0b2800 2023-07-04T18:51:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.212+0000 7fc83bfff700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 0x7fc82806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.213+0000 7fc83bfff700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 0x7fc82806d190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b800 tx=0x7fc830008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.217+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc82c057fa0 con 0x7fc83c0b2800 2023-07-04T18:51:35.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.404+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc83c0ce550 con 0x7fc83c0b2800 2023-07-04T18:51:35.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.404+0000 7fc838ff9700 1 -- 172.21.15.77:0/837792078 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc82c057b30 con 0x7fc83c0b2800 2023-07-04T18:51:35.408 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:35.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 msgr2=0x7fc82806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 0x7fc82806d190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b800 tx=0x7fc830008040 comp rx=0 tx=0).stop 2023-07-04T18:51:35.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 msgr2=0x7fc83c0b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c750 tx=0x7fc82c00cb10 comp rx=0 tx=0).stop 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 shutdown_connections 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc82806ace0 0x7fc82806d190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc83c0cd700 0x7fc83c0b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc83c0b0360 0x7fc83c0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 --2- 172.21.15.77:0/837792078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc83c0b2800 0x7fc83c0b1860 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 >> 172.21.15.77:0/837792078 conn(0x7fc83c06d900 msgr2=0x7fc83c0b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 shutdown_connections 2023-07-04T18:51:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.406+0000 7fc8429cc700 1 -- 172.21.15.77:0/837792078 wait complete. 2023-07-04T18:51:35.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:35.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.541+0000 7fa0235e8700 1 Processor -- start 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa0235e8700 1 -- start start 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa0235e8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa01c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa0235e8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c1606d0 con 0x7fa01c0cd700 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa0235e8700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c0cf210 con 0x7fa01c0b0360 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa0235e8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0cf7d0 con 0x7fa01c0b2800 2023-07-04T18:51:35.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa021b85700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa01c0cd700 0x7fa01c0cc480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51684/0 (socket says 172.21.15.77:51684) 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.542+0000 7fa021b85700 1 -- 172.21.15.77:0/1550968380 learned_addr learned my addr 172.21.15.77:0/1550968380 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687987301 0 0) 0x7fa01c1606d0 con 0x7fa01c0cd700 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa01c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0080035b0 con 0x7fa01c0cd700 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663483825 0 0) 0x7fa0080035b0 con 0x7fa01c0cd700 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 msgr2=0x7fa01c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa01c0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 >> v1:172.21.15.82:6789/0 conn(0x7fa01c0b0360 legacy=0x7fa01c160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0cf4f0 con 0x7fa01c0cd700 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fa0140019f0 con 0x7fa01c0cd700 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 >> v1:172.21.15.77:6789/0 conn(0x7fa01c0cd700 legacy=0x7fa01c0cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa008003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa008003fb0 0x7fa008004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008004d90 0x7fa008005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0cf7d0 con 0x7fa008003fb0 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00800baf0 con 0x7fa008004d90 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00800bcb0 con 0x7fa01c0b2800 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa008003fb0 0x7fa008004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa021384700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa008003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa021b85700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008004d90 0x7fa008005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 -- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 msgr2=0x7fa008003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c0b2800 0x7fa008003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 -- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008004d90 msgr2=0x7fa008005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008004d90 0x7fa008005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.543+0000 7fa020b83700 1 -- 172.21.15.77:0/1550968380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0080035b0 con 0x7fa008003fb0 2023-07-04T18:51:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa020b83700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa008003fb0 0x7fa008004850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0d1c60 tx=0x7fa008005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a13c59be7d6b1f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01800e6c0 con 0x7fa008003fb0 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01800e820 con 0x7fa008003fb0 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa013fff700 1 -- 172.21.15.77:0/1550968380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01800ebb0 con 0x7fa008003fb0 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 -- 172.21.15.77:0/1550968380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa008003fb0 msgr2=0x7fa008004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 --2- 172.21.15.77:0/1550968380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa008003fb0 0x7fa008004850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0d1c60 tx=0x7fa008005880 comp rx=0 tx=0).stop 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa021b85700 1 -- 172.21.15.77:0/1550968380 reap_dead start 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 -- 172.21.15.77:0/1550968380 shutdown_connections 2023-07-04T18:51:35.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 -- 172.21.15.77:0/1550968380 >> 172.21.15.77:0/1550968380 conn(0x7fa01c06d900 msgr2=0x7fa01c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.554 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 -- 172.21.15.77:0/1550968380 shutdown_connections 2023-07-04T18:51:35.554 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.544+0000 7fa0235e8700 1 -- 172.21.15.77:0/1550968380 wait complete. 2023-07-04T18:51:35.554 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.546+0000 7fa0235e8700 1 Processor -- start 2023-07-04T18:51:35.554 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.546+0000 7fa0235e8700 1 -- start start 2023-07-04T18:51:35.556 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 0x7fa01c157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c15e620 0x7fa01c15d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c157e50 con 0x7fa01c0b1780 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0b2130 con 0x7fa008003fb0 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1f35b0 con 0x7fa01c15e620 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44558/0 (socket says 172.21.15.77:44558) 2023-07-04T18:51:35.557 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 -- 172.21.15.77:0/4092533075 learned_addr learned my addr 172.21.15.77:0/4092533075 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021384700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 0x7fa01c157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa020b83700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c15e620 0x7fa01c15d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 -- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c15e620 msgr2=0x7fa01c15d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c15e620 0x7fa01c15d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 -- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 msgr2=0x7fa01c157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 0x7fa01c157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.558 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 -- 172.21.15.77:0/4092533075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c1f2510 con 0x7fa01c0b1780 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021384700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 0x7fa01c157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa021b85700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0cb2d0 tx=0x7fa01400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01400ad00 con 0x7fa01c0b1780 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01400ae60 con 0x7fa01c0b1780 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa01c1f27f0 con 0x7fa01c0b1780 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.547+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014013630 con 0x7fa01c0b1780 2023-07-04T18:51:35.559 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.548+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c1f2db0 con 0x7fa01c0b1780 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.549+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c04e210 con 0x7fa01c0b1780 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.549+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa014013790 con 0x7fa01c0b1780 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.549+0000 7fa011ffb700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 0x7fa00006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.549+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa014089dd0 con 0x7fa01c0b1780 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.549+0000 7fa021384700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 0x7fa00006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.550+0000 7fa021384700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 0x7fa00006d190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007fd0 tx=0x7fa00c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.553+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa014054e80 con 0x7fa01c0b1780 2023-07-04T18:51:35.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.742+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa01c0b3110 con 0x7fa01c0b1780 2023-07-04T18:51:35.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.742+0000 7fa011ffb700 1 -- 172.21.15.77:0/4092533075 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa01408b0a0 con 0x7fa01c0b1780 2023-07-04T18:51:35.745 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:35.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.744+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 msgr2=0x7fa00006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.744+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 0x7fa00006d190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007fd0 tx=0x7fa00c009040 comp rx=0 tx=0).stop 2023-07-04T18:51:35.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 msgr2=0x7fa01c0b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0cb2d0 tx=0x7fa01400a5f0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 shutdown_connections 2023-07-04T18:51:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa00006ace0 0x7fa00006d190 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa01c0b1780 0x7fa01c0b1bf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa008003fb0 0x7fa01c157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 --2- 172.21.15.77:0/4092533075 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa01c15e620 0x7fa01c15d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 >> 172.21.15.77:0/4092533075 conn(0x7fa01c06d900 msgr2=0x7fa01c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 shutdown_connections 2023-07-04T18:51:35.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.745+0000 7fa0235e8700 1 -- 172.21.15.77:0/4092533075 wait complete. 2023-07-04T18:51:35.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:51:35.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.878+0000 7fb5dcf8c700 1 Processor -- start 2023-07-04T18:51:35.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.878+0000 7fb5dcf8c700 1 -- start start 2023-07-04T18:51:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5dcf8c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5d80cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5dcf8c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fb5d80cf920 0x7fb5d806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5dcf8c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80cf100 con 0x7fb5d815a430 2023-07-04T18:51:35.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5dcf8c700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d815df60 con 0x7fb5d80cf920 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5dcf8c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d815e240 con 0x7fb5d815ea40 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d659c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb5d815a430 0x7fb5d80ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51700/0 (socket says 172.21.15.77:51700) 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d659c700 1 -- 172.21.15.77:0/1481473199 learned_addr learned my addr 172.21.15.77:0/1481473199 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> v2:172.21.15.82:3300/0 conn(0x7fb5d80cf920 0x7fb5d806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d5d9b700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5d80cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182474474 0 0) 0x7fb5d80cf100 con 0x7fb5d815a430 2023-07-04T18:51:35.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.879+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c00035b0 con 0x7fb5d815a430 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1859528611 0 0) 0x7fb5c00035b0 con 0x7fb5d815a430 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 msgr2=0x7fb5d80cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5d80cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 >> v2:172.21.15.82:3300/0 conn(0x7fb5d80cf920 msgr2=0x7fb5d806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 --2- 172.21.15.77:0/1481473199 >> v2:172.21.15.82:3300/0 conn(0x7fb5d80cf920 0x7fb5d806ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d815e520 con 0x7fb5d815a430 2023-07-04T18:51:35.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fb5cc001670 con 0x7fb5d815a430 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> v2:172.21.15.82:3300/0 conn(0x7fb5d80cf920 0x7fb5d806ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 >> v1:172.21.15.77:6789/0 conn(0x7fb5d815a430 legacy=0x7fb5d80ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5c0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0003f80 0x7fb5c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5c0004d90 0x7fb5c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d815df60 con 0x7fb5c0004d90 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d815e240 con 0x7fb5c0003f80 2023-07-04T18:51:35.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c000ba10 con 0x7fb5d815ea40 2023-07-04T18:51:35.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5c0004d90 0x7fb5c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d659c700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5c0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d5d9b700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0003f80 0x7fb5c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 msgr2=0x7fb5c0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d815ea40 0x7fb5c0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0003f80 msgr2=0x7fb5c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0003f80 0x7fb5c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/1481473199 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00035b0 con 0x7fb5c0004d90 2023-07-04T18:51:35.892 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5c0004d90 0x7fb5c0005200 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000e350 tx=0x7fb5c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c35d03cf12557e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d000edf0 con 0x7fb5c0004d90 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d000f040 con 0x7fb5c0004d90 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.880+0000 7fb5d559a700 1 -- 172.21.15.77:0/1481473199 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00146a0 con 0x7fb5c0004d90 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/1481473199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5c0004d90 msgr2=0x7fb5c0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/1481473199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5c0004d90 0x7fb5c0005200 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000e350 tx=0x7fb5c0005880 comp rx=0 tx=0).stop 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/1481473199 reap_dead start 2023-07-04T18:51:35.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/1481473199 shutdown_connections 2023-07-04T18:51:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/1481473199 >> 172.21.15.77:0/1481473199 conn(0x7fb5d806d7c0 msgr2=0x7fb5d806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/1481473199 shutdown_connections 2023-07-04T18:51:35.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.881+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/1481473199 wait complete. 2023-07-04T18:51:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.883+0000 7fb5dcf8c700 1 Processor -- start 2023-07-04T18:51:35.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.883+0000 7fb5dcf8c700 1 -- start start 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0004d90 0x7fb5d80b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 0x7fb5d80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d81e4130 con 0x7fb5d81532a0 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d80b2100 con 0x7fb5c0004d90 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d80b0b20 con 0x7fb5d8153a70 2023-07-04T18:51:35.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44578/0 (socket says 172.21.15.77:44578) 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/718967644 learned_addr learned my addr 172.21.15.77:0/718967644 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d659c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0004d90 0x7fb5d80b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d5d9b700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 0x7fb5d80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 msgr2=0x7fb5d80b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 0x7fb5d80b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0004d90 msgr2=0x7fb5d80b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0004d90 0x7fb5d80b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 -- 172.21.15.77:0/718967644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80b0e00 con 0x7fb5d81532a0 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d5d9b700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 0x7fb5d80b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5d6d9d700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00046d0 tx=0x7fb5d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00043e0 con 0x7fb5d81532a0 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d0004560 con 0x7fb5d81532a0 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d80b10e0 con 0x7fb5d81532a0 2023-07-04T18:51:35.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0004910 con 0x7fb5d81532a0 2023-07-04T18:51:35.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.884+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d80b16a0 con 0x7fb5d81532a0 2023-07-04T18:51:35.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.885+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d80ce8c0 con 0x7fb5d81532a0 2023-07-04T18:51:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.886+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb5d0014480 con 0x7fb5d81532a0 2023-07-04T18:51:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.886+0000 7fb5c6ffd700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 0x7fb5b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.886+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb5d008ad90 con 0x7fb5d81532a0 2023-07-04T18:51:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.886+0000 7fb5d659c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 0x7fb5b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:35.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.887+0000 7fb5d659c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 0x7fb5b806d190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc001650 tx=0x7fb5cc007980 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:35.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:35.890+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb5d0055ff0 con 0x7fb5d81532a0 2023-07-04T18:51:36.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.121+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5d804e210 con 0x7fb5d81532a0 2023-07-04T18:51:36.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.121+0000 7fb5c6ffd700 1 -- 172.21.15.77:0/718967644 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7fb5d0018090 con 0x7fb5d81532a0 2023-07-04T18:51:36.123 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:36.123 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:51:36.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 msgr2=0x7fb5b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 0x7fb5b806d190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc001650 tx=0x7fb5cc007980 comp rx=0 tx=0).stop 2023-07-04T18:51:36.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 msgr2=0x7fb5d80b2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00046d0 tx=0x7fb5d0009750 comp rx=0 tx=0).stop 2023-07-04T18:51:36.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 shutdown_connections 2023-07-04T18:51:36.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb5b806ace0 0x7fb5b806d190 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5d81532a0 0x7fb5d80b2730 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5c0004d90 0x7fb5d80b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 --2- 172.21.15.77:0/718967644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb5d8153a70 0x7fb5d80b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 >> 172.21.15.77:0/718967644 conn(0x7fb5d806d7c0 msgr2=0x7fb5d80cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:36.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 shutdown_connections 2023-07-04T18:51:36.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.123+0000 7fb5dcf8c700 1 -- 172.21.15.77:0/718967644 wait complete. 2023-07-04T18:51:36.128 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:51:36.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:36.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.255+0000 7f9515444700 1 Processor -- start 2023-07-04T18:51:36.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.255+0000 7f9515444700 1 -- start start 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f9515444700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f95100b1860 0x7f95100d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f9515444700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f95100d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f9515444700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95100b2790 con 0x7f9510146270 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f9515444700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95100b2a70 con 0x7f95100b1860 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f9515444700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95100b2d50 con 0x7f95100af8f0 2023-07-04T18:51:36.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950f7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9510146270 0x7f95101602d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51722/0 (socket says 172.21.15.77:51722) 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950f7fe700 1 -- 172.21.15.77:0/1272625072 learned_addr learned my addr 172.21.15.77:0/1272625072 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950effd700 1 --2- 172.21.15.77:0/1272625072 >> v2:172.21.15.82:3300/0 conn(0x7f95100b1860 0x7f95100d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669015126 0 0) 0x7f95100b2790 con 0x7f9510146270 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950e7fc700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f95100d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.256+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f80035b0 con 0x7f9510146270 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4131801541 0 0) 0x7f94f80035b0 con 0x7f9510146270 2023-07-04T18:51:36.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 msgr2=0x7f95100d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f95100d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 >> v2:172.21.15.82:3300/0 conn(0x7f95100b1860 msgr2=0x7f95100d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 --2- 172.21.15.77:0/1272625072 >> v2:172.21.15.82:3300/0 conn(0x7f95100b1860 0x7f95100d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100b3030 con 0x7f9510146270 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f95080019f0 con 0x7f9510146270 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 >> v1:172.21.15.77:6789/0 conn(0x7f9510146270 legacy=0x7f95101602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f94f8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f94f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f8004d90 0x7f94f8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100b2a70 con 0x7f94f8003f80 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100b2d50 con 0x7f94f8004d90 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f800ba10 con 0x7f95100af8f0 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950effd700 1 --2- 172.21.15.77:0/1272625072 >> v2:172.21.15.82:3300/0 conn(0x7f95100b1860 0x7f95100d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f94f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950f7fe700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f8004d90 0x7f94f8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950effd700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f94f8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 -- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 msgr2=0x7f94f8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f94f8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 -- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f8004d90 msgr2=0x7f94f8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f8004d90 0x7f94f8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 -- 172.21.15.77:0/1272625072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80035b0 con 0x7f94f8003f80 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950effd700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100af8f0 0x7f94f8003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:36.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950e7fc700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f94f8004850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9500009b60 tx=0x7f94f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8ed5ffacceaca8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000e6c0 con 0x7f94f8003f80 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950000e820 con 0x7f94f8003f80 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.257+0000 7f950dffb700 1 -- 172.21.15.77:0/1272625072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000eb90 con 0x7f94f8003f80 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 -- 172.21.15.77:0/1272625072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 msgr2=0x7f94f8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 --2- 172.21.15.77:0/1272625072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f94f8004850 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9500009b60 tx=0x7f94f8005880 comp rx=0 tx=0).stop 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f950f7fe700 1 -- 172.21.15.77:0/1272625072 reap_dead start 2023-07-04T18:51:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 -- 172.21.15.77:0/1272625072 shutdown_connections 2023-07-04T18:51:36.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 -- 172.21.15.77:0/1272625072 >> 172.21.15.77:0/1272625072 conn(0x7f9510071e90 msgr2=0x7f951009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:36.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 -- 172.21.15.77:0/1272625072 shutdown_connections 2023-07-04T18:51:36.266 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.258+0000 7f9515444700 1 -- 172.21.15.77:0/1272625072 wait complete. 2023-07-04T18:51:36.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.259+0000 7f9515444700 1 Processor -- start 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.259+0000 7f9515444700 1 -- start start 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f95100b5bc0 0x7f9510153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 0x7f9510154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100ce960 con 0x7f94f8003f80 2023-07-04T18:51:36.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100cdd50 con 0x7f95100b5bc0 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f9515444700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95100ce030 con 0x7f95100b6030 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f950effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f950effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44596/0 (socket says 172.21.15.77:44596) 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f950effd700 1 -- 172.21.15.77:0/109149543 learned_addr learned my addr 172.21.15.77:0/109149543 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f950e7fc700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f95100b5bc0 0x7f9510153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.260+0000 7f950f7fe700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 0x7f9510154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 -- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 msgr2=0x7f9510154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 0x7f9510154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 -- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f95100b5bc0 msgr2=0x7f9510153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f95100b5bc0 0x7f9510153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 -- 172.21.15.77:0/109149543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100ce310 con 0x7f94f8003f80 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950f7fe700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 0x7f9510154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:36.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f950effd700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9504010c30 tx=0x7f9504009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400e9c0 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950400eb20 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95100ce5f0 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950400ee90 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.261+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951015af30 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.262+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95100cec40 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.262+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9504016460 con 0x7f94f8003f80 2023-07-04T18:51:36.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.263+0000 7f94f77fe700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 0x7f94fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.263+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9504089ba0 con 0x7f94f8003f80 2023-07-04T18:51:36.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.263+0000 7f950e7fc700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 0x7f94fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.263+0000 7f950e7fc700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 0x7f94fc06d190 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95000090b0 tx=0x7f9500008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.267+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9504058650 con 0x7f94f8003f80 2023-07-04T18:51:36.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.457+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f951004e210 con 0x7f94f8003f80 2023-07-04T18:51:36.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.458+0000 7f94f77fe700 1 -- 172.21.15.77:0/109149543 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f9504024090 con 0x7f94f8003f80 2023-07-04T18:51:36.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:36.460 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:36.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 msgr2=0x7f94fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 0x7f94fc06d190 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95000090b0 tx=0x7f9500008030 comp rx=0 tx=0).stop 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 msgr2=0x7f95100cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9504010c30 tx=0x7f9504009eb0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 shutdown_connections 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f94fc06ace0 0x7f94fc06d190 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94f8003f80 0x7f95100cd4e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f95100b6030 0x7f9510154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 --2- 172.21.15.77:0/109149543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f95100b5bc0 0x7f9510153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 >> 172.21.15.77:0/109149543 conn(0x7f9510071e90 msgr2=0x7f95100afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 shutdown_connections 2023-07-04T18:51:36.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.460+0000 7f9515444700 1 -- 172.21.15.77:0/109149543 wait complete. 2023-07-04T18:51:36.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:51:36.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 Processor -- start 2023-07-04T18:51:36.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 -- start start 2023-07-04T18:51:36.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f52040cf920 0x7f520406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52040cf100 con 0x7f520415a430 2023-07-04T18:51:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520415df60 con 0x7f52040cf920 2023-07-04T18:51:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.595+0000 7f520ac69700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520415e240 con 0x7f520415ea40 2023-07-04T18:51:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5203fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f52040cf920 0x7f520406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5203fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34454/0 (socket says 172.21.15.77:34454) 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5203fff700 1 -- 172.21.15.77:0/4093602606 learned_addr learned my addr 172.21.15.77:0/4093602606 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 --2- 172.21.15.77:0/4093602606 >> v2:172.21.15.82:3300/0 conn(0x7f52040cf920 0x7f520406ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50412/0 (socket says 172.21.15.77:50412) 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 -- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 msgr2=0x7f52040cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 -- 172.21.15.77:0/4093602606 >> v1:172.21.15.82:6789/0 conn(0x7f520415a430 legacy=0x7f52040ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 -- 172.21.15.77:0/4093602606 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520415e520 con 0x7f52040cf920 2023-07-04T18:51:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5203fff700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:36.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f5209206700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f52040cf920 0x7f520406ef20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009d90 tx=0x7f51fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15ad31296438ca62 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.596+0000 7f52037fe700 1 -- 172.21.15.77:0/4093602606 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00e070 con 0x7f52040cf920 2023-07-04T18:51:36.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f52037fe700 1 -- 172.21.15.77:0/4093602606 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc00bb20 con 0x7f52040cf920 2023-07-04T18:51:36.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f52037fe700 1 -- 172.21.15.77:0/4093602606 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc012430 con 0x7f52040cf920 2023-07-04T18:51:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 -- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f52040cf920 msgr2=0x7f520406ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f52040cf920 0x7f520406ef20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009d90 tx=0x7f51fc00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:36.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 -- 172.21.15.77:0/4093602606 shutdown_connections 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f52040cf920 0x7f520406ef20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 --2- 172.21.15.77:0/4093602606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f520415ea40 0x7f52040cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 -- 172.21.15.77:0/4093602606 >> 172.21.15.77:0/4093602606 conn(0x7f520406d7c0 msgr2=0x7f520406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 -- 172.21.15.77:0/4093602606 shutdown_connections 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.597+0000 7f520ac69700 1 -- 172.21.15.77:0/4093602606 wait complete. 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.598+0000 7f520ac69700 1 Processor -- start 2023-07-04T18:51:36.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.598+0000 7f520ac69700 1 -- start start 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f52040cf920 0x7f52041561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f520415ce40 0x7f52041e4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c75d0 con 0x7f520415ea40 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52041f00a0 con 0x7f520415ce40 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f520ac69700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52040c6d30 con 0x7f52040cf920 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f5203fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f5203fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44610/0 (socket says 172.21.15.77:44610) 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f5203fff700 1 -- 172.21.15.77:0/2717153502 learned_addr learned my addr 172.21.15.77:0/2717153502 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f5209206700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f520415ce40 0x7f52041e4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.599+0000 7f5208a05700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f52040cf920 0x7f52041561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 -- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f52040cf920 msgr2=0x7f52041561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f52040cf920 0x7f52041561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 -- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f520415ce40 msgr2=0x7f52041e4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f520415ce40 0x7f52041e4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 -- 172.21.15.77:0/2717153502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0009550 con 0x7f520415ea40 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f5203fff700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f51f000e9b0 tx=0x7f51f000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000ee50 con 0x7f520415ea40 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f000c850 con 0x7f520415ea40 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc009a70 con 0x7f520415ea40 2023-07-04T18:51:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000cbc0 con 0x7f520415ea40 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.600+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52040c72f0 con 0x7f520415ea40 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.601+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520404e210 con 0x7f520415ea40 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.601+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f51f000cd20 con 0x7f520415ea40 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.601+0000 7f52017fa700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 0x7f51e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.602+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f51f00893f0 con 0x7f520415ea40 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.602+0000 7f5208a05700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 0x7f51e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:36.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.602+0000 7f5208a05700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 0x7f51e806d190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f51f8007fd0 tx=0x7f51f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:36.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.606+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f51f00544a0 con 0x7f520415ea40 2023-07-04T18:51:36.835 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.833+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52040c3f60 con 0x7f520415ea40 2023-07-04T18:51:36.836 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.834+0000 7f52017fa700 1 -- 172.21.15.77:0/2717153502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7f51f0057ac0 con 0x7f520415ea40 2023-07-04T18:51:36.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:36.836 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:51:36.837 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 msgr2=0x7f51e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 0x7f51e806d190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f51f8007fd0 tx=0x7f51f8009040 comp rx=0 tx=0).stop 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 msgr2=0x7f52040c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f51f000e9b0 tx=0x7f51f000ecc0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 shutdown_connections 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f51e806ace0 0x7f51e806d190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f520415ce40 0x7f52041e4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f52040cf920 0x7f52041561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 --2- 172.21.15.77:0/2717153502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f520415ea40 0x7f52040c66e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:36.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 >> 172.21.15.77:0/2717153502 conn(0x7f520406d7c0 msgr2=0x7f520415bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:36.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 shutdown_connections 2023-07-04T18:51:36.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:36.836+0000 7f520ac69700 1 -- 172.21.15.77:0/2717153502 wait complete. 2023-07-04T18:51:36.839 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:51:37.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-04T18:51:21.845433+0000', 'modified': '2023-07-04T18:51:25.489990+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4296': {'gid': 4296, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6837/3922191311', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6835', 'nonce': 3922191311}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 3922191311}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-04T18:51:37.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-04T18:51:37.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-04T18:51:37.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.377+0000 7f88424e3700 1 Processor -- start 2023-07-04T18:51:37.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.377+0000 7f88424e3700 1 -- start start 2023-07-04T18:51:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f88424e3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f883c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f88424e3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c071ad0 con 0x7f883c0ce400 2023-07-04T18:51:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f88424e3700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c071d50 con 0x7f883c071670 2023-07-04T18:51:37.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f88424e3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c0cb9f0 con 0x7f883c158c90 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f883b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f883c0ce400 0x7f883c070f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51756/0 (socket says 172.21.15.77:51756) 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f883b7fe700 1 -- 172.21.15.77:0/25027700 learned_addr learned my addr 172.21.15.77:0/25027700 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f883bfff700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f883c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100756170 0 0) 0x7f883c071ad0 con 0x7f883c0ce400 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.378+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88280046c0 con 0x7f883c0ce400 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013726867 0 0) 0x7f883c071d50 con 0x7f883c071670 2023-07-04T18:51:37.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883c071ad0 con 0x7f883c071670 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985765726 0 0) 0x7f88280046c0 con 0x7f883c0ce400 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 msgr2=0x7f883c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f883c0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 >> v1:172.21.15.82:6789/0 conn(0x7f883c071670 legacy=0x7f883c069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c072030 con 0x7f883c0ce400 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f882c001670 con 0x7f883c0ce400 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 >> v1:172.21.15.77:6789/0 conn(0x7f883c0ce400 legacy=0x7f883c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828004cd0 0x7f88280050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8828005960 0x7f8828005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f8828006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c0cb9f0 con 0x7f8828005960 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882800cc00 con 0x7f8828004cd0 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882800cdc0 con 0x7f883c158c90 2023-07-04T18:51:37.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8828005960 0x7f8828005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883bfff700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828004cd0 0x7f88280050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f8840a80700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f8828006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 -- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 msgr2=0x7f8828006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c158c90 0x7f8828006310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 -- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828004cd0 msgr2=0x7f88280050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828004cd0 0x7f88280050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 -- 172.21.15.77:0/25027700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280046c0 con 0x7f8828005960 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883b7fe700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8828005960 0x7f8828005dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c210 tx=0x7f8828006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403d9a04179fd9a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00cd50 con 0x7f8828005960 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c005960 con 0x7f8828005960 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.379+0000 7f883affd700 1 -- 172.21.15.77:0/25027700 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c005d10 con 0x7f8828005960 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 -- 172.21.15.77:0/25027700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8828005960 msgr2=0x7f8828005dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 --2- 172.21.15.77:0/25027700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8828005960 0x7f8828005dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c210 tx=0x7f8828006990 comp rx=0 tx=0).stop 2023-07-04T18:51:37.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f8840a80700 1 -- 172.21.15.77:0/25027700 reap_dead start 2023-07-04T18:51:37.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 -- 172.21.15.77:0/25027700 shutdown_connections 2023-07-04T18:51:37.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 -- 172.21.15.77:0/25027700 >> 172.21.15.77:0/25027700 conn(0x7f883c0cf220 msgr2=0x7f883c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 -- 172.21.15.77:0/25027700 shutdown_connections 2023-07-04T18:51:37.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.380+0000 7f88424e3700 1 -- 172.21.15.77:0/25027700 wait complete. 2023-07-04T18:51:37.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.381+0000 7f88424e3700 1 Processor -- start 2023-07-04T18:51:37.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.381+0000 7f88424e3700 1 -- start start 2023-07-04T18:51:37.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828005960 0x7f883c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 0x7f883c1f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c0c9fd0 con 0x7f883c1f2310 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c154b80 con 0x7f8828005960 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f88424e3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c154e60 con 0x7f883c06afe0 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44632/0 (socket says 172.21.15.77:44632) 2023-07-04T18:51:37.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 -- 172.21.15.77:0/3744838085 learned_addr learned my addr 172.21.15.77:0/3744838085 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:37.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f883bfff700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828005960 0x7f883c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f883b7fe700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 0x7f883c1f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 -- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 msgr2=0x7f883c1f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 0x7f883c1f1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 -- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828005960 msgr2=0x7f883c155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828005960 0x7f883c155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f8840a80700 1 -- 172.21.15.77:0/3744838085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c155140 con 0x7f883c1f2310 2023-07-04T18:51:37.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.382+0000 7f883b7fe700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 0x7f883c1f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f8840a80700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f883c0cd4c0 tx=0x7f883400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f8838ff9700 1 -- 172.21.15.77:0/3744838085 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834003c10 con 0x7f883c1f2310 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f8838ff9700 1 -- 172.21.15.77:0/3744838085 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8834003d70 con 0x7f883c1f2310 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c155420 con 0x7f883c1f2310 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f8838ff9700 1 -- 172.21.15.77:0/3744838085 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883400dbd0 con 0x7f883c1f2310 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.383+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c15bca0 con 0x7f883c1f2310 2023-07-04T18:51:37.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.384+0000 7f8838ff9700 1 -- 172.21.15.77:0/3744838085 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f883400dd30 con 0x7f883c1f2310 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.384+0000 7f8838ff9700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 0x7f881c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.385+0000 7f8838ff9700 1 -- 172.21.15.77:0/3744838085 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8834089740 con 0x7f883c1f2310 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.385+0000 7f883bfff700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 0x7f881c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.385+0000 7f883bfff700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 0x7f881c06d190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8824009b00 tx=0x7f8824009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.386+0000 7f884e668b80 1 Processor -- start 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.386+0000 7f884e668b80 1 -- start start 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5d7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be4b1c60 con 0x5625be5f5160 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be5049f0 con 0x5625be5f4d90 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625be4fde80 con 0x5625be5e8c40 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44640/0 (socket says 172.21.15.77:44640) 2023-07-04T18:51:37.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 -- 172.21.15.77:0/752076610 learned_addr learned my addr 172.21.15.77:0/752076610 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883bfff700 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f8840a80700 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5d7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 -- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 msgr2=0x5625be5d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5d7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 -- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 msgr2=0x5625be5ad4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5ad4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.387+0000 7f883b7fe700 1 -- 172.21.15.77:0/752076610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625be32bd90 con 0x5625be5f5160 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f883b7fe700 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f882800cbd0 tx=0x7f882c009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77e5b0c33208086 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f8831ffb700 1 -- 172.21.15.77:0/752076610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c001630 con 0x5625be5f5160 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f8831ffb700 1 -- 172.21.15.77:0/752076610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c001060 con 0x5625be5f5160 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f8831ffb700 1 -- 172.21.15.77:0/752076610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00cd50 con 0x5625be5f5160 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 -- 172.21.15.77:0/752076610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 msgr2=0x5625be5d7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f882800cbd0 tx=0x7f882c009050 comp rx=0 tx=0).stop 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 -- 172.21.15.77:0/752076610 shutdown_connections 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5d7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5ad4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 --2- 172.21.15.77:0/752076610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f5160 0x5625be5d7030 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 -- 172.21.15.77:0/752076610 >> 172.21.15.77:0/752076610 conn(0x5625be5a1a90 msgr2=0x5625be5a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 -- 172.21.15.77:0/752076610 shutdown_connections 2023-07-04T18:51:37.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.388+0000 7f884e668b80 1 -- 172.21.15.77:0/752076610 wait complete. 2023-07-04T18:51:37.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.389+0000 7f884e668b80 1 -- start start 2023-07-04T18:51:37.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.389+0000 7f884e668b80 1 Processor -- start 2023-07-04T18:51:37.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5e8c40 0x5625be5a7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5f5160 0x5625be5f1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be49da70 con 0x5625be5f4d90 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be49d770 con 0x5625be5e8c40 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625be3818d0 con 0x5625be5f5160 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f884e668b80 10 client.0 _ll_register_callbacks cb 0x5625be49b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-04T18:51:37.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44644/0 (socket says 172.21.15.77:44644) 2023-07-04T18:51:37.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 -- 172.21.15.77:0/1365636051 learned_addr learned my addr 172.21.15.77:0/1365636051 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:37.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883bfff700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5e8c40 0x5625be5a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f8840a80700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5f5160 0x5625be5f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 -- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5f5160 msgr2=0x5625be5f1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5f5160 0x5625be5f1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 -- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5e8c40 msgr2=0x5625be5a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5e8c40 0x5625be5a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.390+0000 7f883b7fe700 1 -- 172.21.15.77:0/1365636051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88280046c0 con 0x5625be5f4d90 2023-07-04T18:51:37.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f882c007040 tx=0x7f882c0019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd3a95784da1c62 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c004870 con 0x5625be5f4d90 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f882c0049d0 con 0x5625be5f4d90 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625be59f0a0 con 0x5625be5f4d90 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f882c004b90 con 0x5625be5f4d90 2023-07-04T18:51:37.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 10 client.4680 fetch_fsmap learned FSMap version 8 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5625be314c10 con 0x5625be5f4d90 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5625be59f0a0 con 0x5625be5f4d90 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f882c00c240 con 0x5625be5f4d90 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 10 client.4680 fetch_fsmap finished waiting for FSMap version 8 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 10 client.4680 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-04T18:51:37.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 20 client.4680 populate_metadata read hostname 'smithi077' 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5ec660 0x5625be5eca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 4 client.4680 mds_command: new command op to 4296 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 --> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625be4f6e60 con 0x5625be5ec660 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.391+0000 7f883bfff700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5ec660 0x5625be5eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.392+0000 7f881affd700 10 client.4680 ms_handle_connect on v2:172.21.15.82:6835/3922191311 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.392+0000 7f883bfff700 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5ec660 0x5625be5eca30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c776eeb1856a9a server_cookie=1b0db681d65ba4fb in_seq=0 out_seq=0 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.392+0000 7f881affd700 1 -- 172.21.15.77:0/1365636051 <== mds.0 v2:172.21.15.82:6835/3922191311 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5625be4f6e60 con 0x5625be5ec660 2023-07-04T18:51:37.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.392+0000 7f881affd700 10 client.4680 handle_command_reply: tid=0 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.392+0000 7f884e668b80 1 client.4680 shutdown 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 msgr2=0x5625be5f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f882c007040 tx=0x7f882c0019f0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 shutdown_connections 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5ec660 0x5625be5eca30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5f5160 0x5625be5f1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f881affd700 0 client.4680 ms_handle_reset on v2:172.21.15.82:6835/3922191311 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5e8c40 0x5625be5a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 --2- 172.21.15.77:0/1365636051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5f4d90 0x5625be5f0d00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 >> 172.21.15.77:0/1365636051 conn(0x5625be5a1a90 msgr2=0x5625be5d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 shutdown_connections 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 1 -- 172.21.15.77:0/1365636051 wait complete. 2023-07-04T18:51:37.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.393+0000 7f884e668b80 20 client.4680 trim_cache size 0 max 16384 2023-07-04T18:51:37.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.410+0000 7f884e668b80 1 Processor -- start 2023-07-04T18:51:37.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.410+0000 7f884e668b80 1 -- start start 2023-07-04T18:51:37.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5a8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be5a7d40 con 0x5625be5ec660 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be5eeb70 con 0x5625be5f4d90 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625be613a80 con 0x5625be5e8c40 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44652/0 (socket says 172.21.15.77:44652) 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 -- 172.21.15.77:0/1109304008 learned_addr learned my addr 172.21.15.77:0/1109304008 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f8840a80700 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883bfff700 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5a8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 -- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 msgr2=0x5625be5a8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5a8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 -- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 msgr2=0x5625be5e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 -- 172.21.15.77:0/1109304008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625be352200 con 0x5625be5ec660 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c8d0 tx=0x7f882c02a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6884a0579464d33f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f88327fc700 1 -- 172.21.15.77:0/1109304008 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c005ce0 con 0x5625be5ec660 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f88327fc700 1 -- 172.21.15.77:0/1109304008 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c005e40 con 0x5625be5ec660 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.411+0000 7f88327fc700 1 -- 172.21.15.77:0/1109304008 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c004a90 con 0x5625be5ec660 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- 172.21.15.77:0/1109304008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 msgr2=0x5625be5f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f882c00c8d0 tx=0x7f882c02a040 comp rx=0 tx=0).stop 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- 172.21.15.77:0/1109304008 shutdown_connections 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5e8c40 0x5625be5a8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 --2- 172.21.15.77:0/1109304008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5ec660 0x5625be5f0cb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- 172.21.15.77:0/1109304008 >> 172.21.15.77:0/1109304008 conn(0x5625be5a1a90 msgr2=0x5625be5dabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- 172.21.15.77:0/1109304008 shutdown_connections 2023-07-04T18:51:37.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- 172.21.15.77:0/1109304008 wait complete. 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.412+0000 7f884e668b80 1 -- start start 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 Processor -- start 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5ec660 0x5625be5e83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be658020 con 0x5625be5e8c40 2023-07-04T18:51:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625be5f2320 con 0x5625be5f4d90 2023-07-04T18:51:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625be5a9000 con 0x5625be5ec660 2023-07-04T18:51:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f884e668b80 10 client.0 _ll_register_callbacks cb 0x7f88240097f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-04T18:51:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f883bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f883bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44656/0 (socket says 172.21.15.77:44656) 2023-07-04T18:51:37.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.413+0000 7f883bfff700 1 -- 172.21.15.77:0/1238311702 learned_addr learned my addr 172.21.15.77:0/1238311702 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f8840a80700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5ec660 0x5625be5e83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 -- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5ec660 msgr2=0x5625be5e83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5ec660 0x5625be5e83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 -- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 msgr2=0x5625be5d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 -- 172.21.15.77:0/1238311702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625be314c10 con 0x5625be5e8c40 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f883bfff700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8824009f70 tx=0x7f8824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3ea34bf5ad768 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824027070 con 0x5625be5e8c40 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f882402b2a0 con 0x5625be5e8c40 2023-07-04T18:51:37.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f882c001060 con 0x5625be5e8c40 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f882402b670 con 0x5625be5e8c40 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 10 client.4686 fetch_fsmap learned FSMap version 8 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x5625be32a260 con 0x5625be5e8c40 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882c001060 con 0x5625be5e8c40 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f882402b7d0 con 0x5625be5e8c40 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 10 client.4686 fetch_fsmap finished waiting for FSMap version 8 2023-07-04T18:51:37.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 10 client.4686 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 20 client.4686 populate_metadata read hostname 'smithi077' 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5f5160 0x5625be5d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 4 client.4686 mds_command: new command op to 4296 tid=0[{"prefix": "status"}] 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.414+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 --> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5625be4b9000 con 0x5625be5f5160 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5f5160 0x5625be5d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f881affd700 10 client.4686 ms_handle_connect on v2:172.21.15.82:6835/3922191311 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f883b7fe700 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5f5160 0x5625be5d1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9288639e269d998 server_cookie=60f9fda0f50e1d9c in_seq=0 out_seq=0 2023-07-04T18:51:37.434 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f881affd700 1 -- 172.21.15.77:0/1238311702 <== mds.0 v2:172.21.15.82:6835/3922191311 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x5625be4b9000 con 0x5625be5f5160 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f881affd700 10 client.4686 handle_command_reply: tid=0 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 client.4686 shutdown 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 msgr2=0x5625be5a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8824009f70 tx=0x7f8824004750 comp rx=0 tx=0).stop 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 shutdown_connections 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:6835/3922191311,v1:172.21.15.82:6837/3922191311] conn(0x5625be5f5160 0x5625be5d1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5625be5e8c40 0x5625be5a6630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f881affd700 0 client.4686 ms_handle_reset on v2:172.21.15.82:6835/3922191311 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625be5f4d90 0x5625be5d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 --2- 172.21.15.77:0/1238311702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5625be5ec660 0x5625be5e83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 >> 172.21.15.77:0/1238311702 conn(0x5625be5a1a90 msgr2=0x5625be5d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 shutdown_connections 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 1 -- 172.21.15.77:0/1238311702 wait complete. 2023-07-04T18:51:37.436 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.415+0000 7f884e668b80 20 client.4686 trim_cache size 0 max 16384 2023-07-04T18:51:37.437 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-07-04T18:51:37.437 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5", 2023-07-04T18:51:37.437 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2023-07-04T18:51:37.437 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4296, 2023-07-04T18:51:37.437 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 14.93839944, 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 8, 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 23, 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 23, 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 19.992415178000002 2023-07-04T18:51:37.438 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-07-04T18:51:37.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 msgr2=0x7f881c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 0x7f881c06d190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8824009b00 tx=0x7f8824009210 comp rx=0 tx=0).stop 2023-07-04T18:51:37.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 msgr2=0x7f883c159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:37.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f883c0cd4c0 tx=0x7f883400d430 comp rx=0 tx=0).stop 2023-07-04T18:51:37.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 shutdown_connections 2023-07-04T18:51:37.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f881c06ace0 0x7f881c06d190 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f883c1f2310 0x7f883c159440 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.441 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8828005960 0x7f883c155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.442 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 --2- 172.21.15.77:0/3744838085 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f883c06afe0 0x7f883c1f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:37.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 >> 172.21.15.77:0/3744838085 conn(0x7f883c0cf220 msgr2=0x7f883c065bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:37.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 shutdown_connections 2023-07-04T18:51:37.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:37.416+0000 7f88424e3700 1 -- 172.21.15.77:0/3744838085 wait complete. 2023-07-04T18:51:37.443 INFO:teuthology.run_tasks:Running task kclient... 2023-07-04T18:51:37.470 INFO:tasks.kclient:Mounting kernel clients... 2023-07-04T18:51:37.471 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-07-04T18:51:37.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.472 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2023-07-04T18:51:37.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2023-07-04T18:51:37.556 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2023-07-04T18:51:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:51:37.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.559 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2023-07-04T18:51:37.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2023-07-04T18:51:37.682 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2023-07-04T18:51:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:51:37.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.684 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list 2023-07-04T18:51:37.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link delete ceph-brx 2023-07-04T18:51:37.777 INFO:teuthology.orchestra.run.smithi082.stderr:Cannot find device "ceph-brx" 2023-07-04T18:51:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:51:37.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.779 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list 2023-07-04T18:51:37.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:37.839 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link delete ceph-brx 2023-07-04T18:51:37.908 INFO:teuthology.orchestra.run.smithi082.stderr:Cannot find device "ceph-brx" 2023-07-04T18:51:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T18:51:37.910 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-07-04T18:51:37.910 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-04T18:51:37.910 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-04T18:51:37.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-07-04T18:51:38.030 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.029+0000 7f8613fc6700 1 Processor -- start 2023-07-04T18:51:38.030 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.029+0000 7f8613fc6700 1 -- start start 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8613fc6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c15ea40 0x7f860c0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8613fc6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f860c0cf920 0x7f860c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8613fc6700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0cf100 con 0x7f860c15a430 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8613fc6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c15df60 con 0x7f860c0cf920 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8613fc6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c15e240 con 0x7f860c15ea40 2023-07-04T18:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f860c0cf920 0x7f860c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f860c0cf920 0x7f860c06ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44668/0 (socket says 172.21.15.77:44668) 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 -- 172.21.15.77:0/427252994 learned_addr learned my addr 172.21.15.77:0/427252994 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 -- 172.21.15.77:0/427252994 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c15ea40 msgr2=0x7f860c0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 --2- 172.21.15.77:0/427252994 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c15ea40 0x7f860c0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 -- 172.21.15.77:0/427252994 >> v1:172.21.15.82:6789/0 conn(0x7f860c15a430 legacy=0x7f860c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 -- 172.21.15.77:0/427252994 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c15e520 con 0x7f860c0cf920 2023-07-04T18:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8612563700 1 --2- 172.21.15.77:0/427252994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c06ef20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8604009e50 tx=0x7f860400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb03a4eb6c85e87 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8610d60700 1 -- 172.21.15.77:0/427252994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860400e070 con 0x7f860c0cf920 2023-07-04T18:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8610d60700 1 -- 172.21.15.77:0/427252994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8604003a20 con 0x7f860c0cf920 2023-07-04T18:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.030+0000 7f8610d60700 1 -- 172.21.15.77:0/427252994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604003dd0 con 0x7f860c0cf920 2023-07-04T18:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 -- 172.21.15.77:0/427252994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 msgr2=0x7f860c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 --2- 172.21.15.77:0/427252994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c06ef20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8604009e50 tx=0x7f860400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 -- 172.21.15.77:0/427252994 shutdown_connections 2023-07-04T18:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 --2- 172.21.15.77:0/427252994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c06ef20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 --2- 172.21.15.77:0/427252994 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c15ea40 0x7f860c0cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 -- 172.21.15.77:0/427252994 >> 172.21.15.77:0/427252994 conn(0x7f860c06d7c0 msgr2=0x7f860c06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 -- 172.21.15.77:0/427252994 shutdown_connections 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.031+0000 7f8613fc6700 1 -- 172.21.15.77:0/427252994 wait complete. 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.032+0000 7f8613fc6700 1 Processor -- start 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.032+0000 7f8613fc6700 1 -- start start 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 0x7f860c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 0x7f860c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c4e00 con 0x7f860c0cf920 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c6260 con 0x7f860c15ea40 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8613fc6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c4b30 con 0x7f860c0c72c0 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44670/0 (socket says 172.21.15.77:44670) 2023-07-04T18:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 -- 172.21.15.77:0/885055295 learned_addr learned my addr 172.21.15.77:0/885055295 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611561700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 0x7f860c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8612563700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 0x7f860c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 -- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 msgr2=0x7f860c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 0x7f860c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 -- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 msgr2=0x7f860c0b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 0x7f860c0b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 -- 172.21.15.77:0/885055295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604009b30 con 0x7f860c0cf920 2023-07-04T18:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8612563700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 0x7f860c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611561700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 0x7f860c0b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f8611d62700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f860800c4a0 tx=0x7f860800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.033+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86080054e0 con 0x7f860c0cf920 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.034+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8608003680 con 0x7f860c0cf920 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.034+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8608003a30 con 0x7f860c0cf920 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.034+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c0c6500 con 0x7f860c0cf920 2023-07-04T18:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.034+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c0c6ac0 con 0x7f860c0cf920 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.034+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c15a9a0 con 0x7f860c0cf920 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.035+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8608003b90 con 0x7f860c0cf920 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.035+0000 7f86027fc700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 0x7f85f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.035+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8608088590 con 0x7f860c0cf920 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.035+0000 7f8611561700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 0x7f85f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.036+0000 7f8611561700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 0x7f85f006d190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85fc006fd0 tx=0x7f85fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:38.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.040+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8608053640 con 0x7f860c0cf920 2023-07-04T18:51:38.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.269+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f860c04e210 con 0x7f860c0cf920 2023-07-04T18:51:38.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.269+0000 7f86027fc700 1 -- 172.21.15.77:0/885055295 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8608056c60 con 0x7f860c0cf920 2023-07-04T18:51:38.271 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-04T18:51:38.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 msgr2=0x7f85f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:38.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 0x7f85f006d190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85fc006fd0 tx=0x7f85fc008040 comp rx=0 tx=0).stop 2023-07-04T18:51:38.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 msgr2=0x7f860c0c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:38.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f860800c4a0 tx=0x7f860800c860 comp rx=0 tx=0).stop 2023-07-04T18:51:38.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 shutdown_connections 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f85f006ace0 0x7f85f006d190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f860c0cf920 0x7f860c0c45f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f860c0c72c0 0x7f860c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 --2- 172.21.15.77:0/885055295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f860c15ea40 0x7f860c0b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 >> 172.21.15.77:0/885055295 conn(0x7f860c06d7c0 msgr2=0x7f860c156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 shutdown_connections 2023-07-04T18:51:38.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:38.271+0000 7f8613fc6700 1 -- 172.21.15.77:0/885055295 wait complete. 2023-07-04T18:51:38.284 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-04T18:51:38.285 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-04T18:51:38.285 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2023-07-04T18:51:38.285 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-04T18:51:38.285 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-04T18:51:38.286 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-04T18:51:38.286 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-04T18:51:38.286 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-04T18:51:38.286 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-04T18:51:38.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.287 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2023-07-04T18:51:38.304 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:38.305 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-04T18:51:38.306 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:38.307 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-04T18:51:38.307 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42171sec preferred_lft 42171sec 2023-07-04T18:51:38.307 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8178:5374:91c4:639d/64 scope link noprefixroute 2023-07-04T18:51:38.307 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:38.307 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-04T18:51:38.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.308 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:38.308 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:38.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2023-07-04T18:51:38.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2023-07-04T18:51:38.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2023-07-04T18:51:38.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-04T18:51:38.309 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:38.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.509 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-04T18:51:38.538 INFO:teuthology.orchestra.run.smithi077.stdout:1 2023-07-04T18:51:38.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.539 DEBUG:teuthology.orchestra.run.smithi077:> route 2023-07-04T18:51:38.607 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2023-07-04T18:51:38.608 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-04T18:51:38.608 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-04T18:51:38.608 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-04T18:51:38.608 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-04T18:51:38.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.609 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:38.609 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:38.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-04T18:51:38.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-04T18:51:38.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-04T18:51:38.610 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:38.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.860 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2023-07-04T18:51:38.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.878 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2023-07-04T18:51:38.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:38.935 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:38.936 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:38.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-04T18:51:38.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-04T18:51:38.937 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:39.028 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-04T18:51:39.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.030 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:39.030 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:39.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-04T18:51:39.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-04T18:51:39.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-04T18:51:39.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-04T18:51:39.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-04T18:51:39.033 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:39.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.273 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:39.273 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:39.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2023-07-04T18:51:39.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2023-07-04T18:51:39.274 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:39.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:39.377 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-04T18:51:39.396 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-04T18:51:39.397 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:39.397 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-04T18:51:39.465 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-07-04T18:51:39.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.466 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:39.557 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:39.557 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:39.660 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:39.661 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:39.661 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:39.661 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:39.662 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:39.662 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:39.662 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:39.663 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:39.746 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:39.746 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:39.746 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:39.746 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790" 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790:/ 2023-07-04T18:51:39.747 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5" will pass to kernel 2023-07-04T18:51:39.748 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-04T18:51:39.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-04T18:51:39.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.778 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:39.866 INFO:teuthology.orchestra.run.smithi077.stdout:client4502 2023-07-04T18:51:39.866 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:39.867 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:39.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.885 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:39.972 INFO:teuthology.orchestra.run.smithi077.stdout:client4502 2023-07-04T18:51:39.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:39.973 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:40.024 INFO:teuthology.orchestra.run.smithi077.stdout:client4502 2023-07-04T18:51:40.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.025 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.client4502/status') 2023-07-04T18:51:40.078 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4502 (3)172.21.15.77:0/4104520017 2023-07-04T18:51:40.079 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2023-07-04T18:51:40.080 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2023-07-04T18:51:40.080 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2023-07-04T18:51:40.080 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 7.3931e-05 s, 947 kB/s 2023-07-04T18:51:40.081 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-07-04T18:51:40.083 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-04T18:51:40.084 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-07-04T18:51:40.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-07-04T18:51:40.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.203+0000 7fde4ba55700 1 Processor -- start 2023-07-04T18:51:40.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.203+0000 7fde4ba55700 1 -- start start 2023-07-04T18:51:40.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde4ba55700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde44158cc0 0x7fde4406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde4ba55700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde4406cbe0 0x7fde440710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde4ba55700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde44071620 con 0x7fde440cb230 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde4ba55700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde440cdd70 con 0x7fde4406cbe0 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde4ba55700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde440ce050 con 0x7fde44158cc0 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde4406cbe0 0x7fde440710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde4406cbe0 0x7fde440710e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44684/0 (socket says 172.21.15.77:44684) 2023-07-04T18:51:40.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 -- 172.21.15.77:0/1679427381 learned_addr learned my addr 172.21.15.77:0/1679427381 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde497f1700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde44158cc0 0x7fde4406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 -- 172.21.15.77:0/1679427381 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde44158cc0 msgr2=0x7fde4406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde44158cc0 0x7fde4406f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 -- 172.21.15.77:0/1679427381 >> v1:172.21.15.82:6789/0 conn(0x7fde440cb230 legacy=0x7fde440cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 -- 172.21.15.77:0/1679427381 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde440ce330 con 0x7fde4406cbe0 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde48ff0700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde4406cbe0 0x7fde440710e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde34009920 tx=0x7fde34009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e3b8ca6f636431 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:40.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.204+0000 7fde3bfff700 1 -- 172.21.15.77:0/1679427381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3400ba20 con 0x7fde4406cbe0 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde3bfff700 1 -- 172.21.15.77:0/1679427381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3400bb80 con 0x7fde4406cbe0 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde3bfff700 1 -- 172.21.15.77:0/1679427381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34009e70 con 0x7fde4406cbe0 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 -- 172.21.15.77:0/1679427381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde4406cbe0 msgr2=0x7fde440710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde4406cbe0 0x7fde440710e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde34009920 tx=0x7fde34009c30 comp rx=0 tx=0).stop 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 -- 172.21.15.77:0/1679427381 shutdown_connections 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde44158cc0 0x7fde4406f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1679427381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde4406cbe0 0x7fde440710e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 -- 172.21.15.77:0/1679427381 >> 172.21.15.77:0/1679427381 conn(0x7fde4406aad0 msgr2=0x7fde440b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:40.210 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 -- 172.21.15.77:0/1679427381 shutdown_connections 2023-07-04T18:51:40.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.205+0000 7fde4ba55700 1 -- 172.21.15.77:0/1679427381 wait complete. 2023-07-04T18:51:40.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.206+0000 7fde4ba55700 1 Processor -- start 2023-07-04T18:51:40.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.206+0000 7fde4ba55700 1 -- start start 2023-07-04T18:51:40.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 0x7fde440c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 0x7fde44159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde440c5ec0 con 0x7fde440cb230 2023-07-04T18:51:40.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde440b35e0 con 0x7fde44158cc0 2023-07-04T18:51:40.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde4ba55700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde44149090 con 0x7fde4406cbe0 2023-07-04T18:51:40.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde48ff0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde48ff0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44692/0 (socket says 172.21.15.77:44692) 2023-07-04T18:51:40.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde48ff0700 1 -- 172.21.15.77:0/1744734218 learned_addr learned my addr 172.21.15.77:0/1744734218 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:40.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde497f1700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 0x7fde440c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde49ff2700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 0x7fde44159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.207+0000 7fde48ff0700 1 -- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 msgr2=0x7fde440c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:40.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde48ff0700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 0x7fde440c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde48ff0700 1 -- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 msgr2=0x7fde44159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:40.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde48ff0700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 0x7fde44159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde48ff0700 1 -- 172.21.15.77:0/1744734218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34009550 con 0x7fde440cb230 2023-07-04T18:51:40.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde497f1700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 0x7fde440c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:40.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde49ff2700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 0x7fde44159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:40.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde48ff0700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde34000c00 tx=0x7fde34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:40.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34004400 con 0x7fde440cb230 2023-07-04T18:51:40.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3401f070 con 0x7fde440cb230 2023-07-04T18:51:40.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4415bf40 con 0x7fde440cb230 2023-07-04T18:51:40.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde340038e0 con 0x7fde440cb230 2023-07-04T18:51:40.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.208+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde440b2f90 con 0x7fde440cb230 2023-07-04T18:51:40.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.209+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde440b3790 con 0x7fde440cb230 2023-07-04T18:51:40.222 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.209+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fde34021030 con 0x7fde440cb230 2023-07-04T18:51:40.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.210+0000 7fde39ffb700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 0x7fde2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:40.223 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.210+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fde3408a880 con 0x7fde440cb230 2023-07-04T18:51:40.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.210+0000 7fde497f1700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 0x7fde2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:40.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.210+0000 7fde497f1700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 0x7fde2806d190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde40009820 tx=0x7fde40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:40.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.214+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde34055ae0 con 0x7fde440cb230 2023-07-04T18:51:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.442+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fde4404e210 con 0x7fde440cb230 2023-07-04T18:51:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.443+0000 7fde39ffb700 1 -- 172.21.15.77:0/1744734218 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fde34015070 con 0x7fde440cb230 2023-07-04T18:51:40.445 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-04T18:51:40.446 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 msgr2=0x7fde2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 0x7fde2806d190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde40009820 tx=0x7fde40008040 comp rx=0 tx=0).stop 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 msgr2=0x7fde44148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde34000c00 tx=0x7fde34004750 comp rx=0 tx=0).stop 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 shutdown_connections 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fde2806ace0 0x7fde2806d190 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde44158cc0 0x7fde44159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fde4406cbe0 0x7fde440c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 --2- 172.21.15.77:0/1744734218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde440cb230 0x7fde44148b50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:40.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 >> 172.21.15.77:0/1744734218 conn(0x7fde4406aad0 msgr2=0x7fde4414f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:40.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 shutdown_connections 2023-07-04T18:51:40.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:40.445+0000 7fde4ba55700 1 -- 172.21.15.77:0/1744734218 wait complete. 2023-07-04T18:51:40.458 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-04T18:51:40.459 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-04T18:51:40.460 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2023-07-04T18:51:40.460 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-07-04T18:51:40.460 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-07-04T18:51:40.461 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-04T18:51:40.461 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-04T18:51:40.462 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-04T18:51:40.462 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-07-04T18:51:40.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.464 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2023-07-04T18:51:40.481 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:40.481 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:40.481 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:40.481 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:40.482 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42169sec preferred_lft 42169sec 2023-07-04T18:51:40.483 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8178:5374:91c4:639d/64 scope link noprefixroute 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 5e:ca:c7:f8:96:fc brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ac9c:d0ff:fedf:fce1/64 scope link 2023-07-04T18:51:40.484 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.485 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-07-04T18:51:40.485 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 5e:ca:c7:f8:96:fc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-04T18:51:40.485 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::5cca:c7ff:fef8:96fc/64 scope link tentative 2023-07-04T18:51:40.485 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.486 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-04T18:51:40.546 INFO:teuthology.orchestra.run.smithi077.stdout:1 2023-07-04T18:51:40.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.551 DEBUG:teuthology.orchestra.run.smithi077:> route 2023-07-04T18:51:40.619 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2023-07-04T18:51:40.619 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-04T18:51:40.619 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-04T18:51:40.621 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-04T18:51:40.621 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-04T18:51:40.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.622 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:40.622 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:40.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-04T18:51:40.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-04T18:51:40.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-04T18:51:40.627 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:40.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.725 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2023-07-04T18:51:40.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-04T18:51:40.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.749 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2023-07-04T18:51:40.809 INFO:teuthology.orchestra.run.smithi077.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-07-04T18:51:40.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.810 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:40.810 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:40.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-07-04T18:51:40.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-07-04T18:51:40.811 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:40.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.904 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-07-04T18:51:40.942 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:40.942 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 02:13:5d:5b:3b:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-07-04T18:51:40.943 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-07-04T18:51:40.944 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.944 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::13:5dff:fe5b:3b1d/64 scope link 2023-07-04T18:51:40.944 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:40.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:40.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-07-04T18:51:41.038 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:41.039 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:41.040 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:41.040 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:41.041 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:41.041 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:41.041 INFO:teuthology.orchestra.run.smithi077.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:41.042 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 02:13:5d:5b:3b:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-07-04T18:51:41.042 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-07-04T18:51:41.042 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:41.043 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::13:5dff:fe5b:3b1d/64 scope link 2023-07-04T18:51:41.043 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:41.047 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-07-04T18:51:41.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.048 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:41.049 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:41.049 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-07-04T18:51:41.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-07-04T18:51:41.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-07-04T18:51:41.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-07-04T18:51:41.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-07-04T18:51:41.051 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:41.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.305 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:51:41.305 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:51:41.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.1 up 2023-07-04T18:51:41.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.1 master ceph-brx 2023-07-04T18:51:41.307 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:51:41.442 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:41.443 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-07-04T18:51:41.463 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-07-04T18:51:41.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:41.463 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-07-04T18:51:41.520 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-07-04T18:51:41.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:41.606 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:41.607 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:41.634 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 2@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:41.635 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:41.636 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 2@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:41.711 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790" 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790:/ 2023-07-04T18:51:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,key=2,mds_namespace=cephfs,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5" will pass to kernel 2023-07-04T18:51:41.713 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-04T18:51:41.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-07-04T18:51:41.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.741 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:41.813 INFO:teuthology.orchestra.run.smithi077.stdout:client4588 2023-07-04T18:51:41.813 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:41.813 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:41.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.872 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:41.944 INFO:teuthology.orchestra.run.smithi077.stdout:client4588 2023-07-04T18:51:41.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:41.945 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:42.017 INFO:teuthology.orchestra.run.smithi077.stdout:client4588 2023-07-04T18:51:42.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.017 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.client4588/status') 2023-07-04T18:51:42.097 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.4588 (3)172.21.15.77:0/3181877607 2023-07-04T18:51:42.098 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2023-07-04T18:51:42.099 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2023-07-04T18:51:42.099 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2023-07-04T18:51:42.100 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 2.0381e-05 s, 3.4 MB/s 2023-07-04T18:51:42.100 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-07-04T18:51:42.100 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-04T18:51:42.101 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-07-04T18:51:42.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-07-04T18:51:42.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 Processor -- start 2023-07-04T18:51:42.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 -- start start 2023-07-04T18:51:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fa8bc146270 0x7fa8bc0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc0b2790 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8bc0b2a70 con 0x7fa8bc146270 2023-07-04T18:51:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.259+0000 7fa8c1628700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0b2d50 con 0x7fa8bc0b1860 2023-07-04T18:51:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fa8bc146270 0x7fa8bc0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:34554/0 (socket says 172.21.15.77:34554) 2023-07-04T18:51:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8baffd700 1 -- 172.21.15.77:0/4294353923 learned_addr learned my addr 172.21.15.77:0/4294353923 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 --2- 172.21.15.77:0/4294353923 >> v2:172.21.15.82:3300/0 conn(0x7fa8bc146270 0x7fa8bc0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50514/0 (socket says 172.21.15.77:50514) 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8ba7fc700 1 --1- 172.21.15.77:0/4294353923 >> v1:172.21.15.82:6789/0 conn(0x7fa8bc0af8f0 0x7fa8bc1602d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59754/0 (socket says 172.21.15.77:59754) 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 -- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 msgr2=0x7fa8bc0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 -- 172.21.15.77:0/4294353923 >> v1:172.21.15.82:6789/0 conn(0x7fa8bc0af8f0 legacy=0x7fa8bc1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 -- 172.21.15.77:0/4294353923 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0b3030 con 0x7fa8bc146270 2023-07-04T18:51:42.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8baffd700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8bb7fe700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0d31f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009d90 tx=0x7fa8b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=810f2a0610e587a9 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8b9ffb700 1 -- 172.21.15.77:0/4294353923 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400e070 con 0x7fa8bc146270 2023-07-04T18:51:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8b9ffb700 1 -- 172.21.15.77:0/4294353923 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b400bb20 con 0x7fa8bc146270 2023-07-04T18:51:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.260+0000 7fa8b9ffb700 1 -- 172.21.15.77:0/4294353923 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4012430 con 0x7fa8bc146270 2023-07-04T18:51:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 -- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 msgr2=0x7fa8bc0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0d31f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009d90 tx=0x7fa8b400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 -- 172.21.15.77:0/4294353923 shutdown_connections 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0d31f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 --2- 172.21.15.77:0/4294353923 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 -- 172.21.15.77:0/4294353923 >> 172.21.15.77:0/4294353923 conn(0x7fa8bc071e90 msgr2=0x7fa8bc09c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 -- 172.21.15.77:0/4294353923 shutdown_connections 2023-07-04T18:51:42.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.261+0000 7fa8c1628700 1 -- 172.21.15.77:0/4294353923 wait complete. 2023-07-04T18:51:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.262+0000 7fa8c1628700 1 Processor -- start 2023-07-04T18:51:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.262+0000 7fa8c1628700 1 -- start start 2023-07-04T18:51:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc1f1d40 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0cb520 con 0x7fa8bc146270 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8c1628700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0cb800 con 0x7fa8bc0b1860 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44710/0 (socket says 172.21.15.77:44710) 2023-07-04T18:51:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8baffd700 1 -- 172.21.15.77:0/732515147 learned_addr learned my addr 172.21.15.77:0/732515147 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.263+0000 7fa8ba7fc700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8bb7fe700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 -- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 msgr2=0x7fa8bc0c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 -- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 msgr2=0x7fa8bc0c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0c7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 -- 172.21.15.77:0/732515147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0009550 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8baffd700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000ebe0 tx=0x7fa8b0009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000c8b0 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b000ca10 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b4009a70 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000cd80 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.264+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc0cbdc0 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.265+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8bc0cbfe0 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.265+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa8b0013470 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.266+0000 7fa8a37fe700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 0x7fa8a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.266+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa8b00895c0 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.266+0000 7fa8ba7fc700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 0x7fa8a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.266+0000 7fa8ba7fc700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 0x7fa8a806d190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0cff50 tx=0x7fa8ac007d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.270+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa8b0054700 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.506+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa8bc15db00 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.506+0000 7fa8a37fe700 1 -- 172.21.15.77:0/732515147 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa8b0057d20 con 0x7fa8bc0af8f0 2023-07-04T18:51:42.508 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-04T18:51:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 msgr2=0x7fa8a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 0x7fa8a806d190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0cff50 tx=0x7fa8ac007d10 comp rx=0 tx=0).stop 2023-07-04T18:51:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 msgr2=0x7fa8bc1f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000ebe0 tx=0x7fa8b0009e70 comp rx=0 tx=0).stop 2023-07-04T18:51:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 shutdown_connections 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8a806ace0 0x7fa8a806d190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8bc0af8f0 0x7fa8bc1f1800 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc146270 0x7fa8bc0c7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 --2- 172.21.15.77:0/732515147 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa8bc0b1860 0x7fa8bc0c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 >> 172.21.15.77:0/732515147 conn(0x7fa8bc071e90 msgr2=0x7fa8bc0afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 shutdown_connections 2023-07-04T18:51:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:42.508+0000 7fa8c1628700 1 -- 172.21.15.77:0/732515147 wait complete. 2023-07-04T18:51:42.522 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-04T18:51:42.523 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-04T18:51:42.523 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi082.front.sepia.ceph.com 2023-07-04T18:51:42.524 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-07-04T18:51:42.524 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-07-04T18:51:42.525 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-04T18:51:42.525 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-04T18:51:42.525 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-04T18:51:42.526 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-07-04T18:51:42.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.527 DEBUG:teuthology.orchestra.run.smithi082:> ip addr 2023-07-04T18:51:42.549 INFO:teuthology.orchestra.run.smithi082.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:42.550 INFO:teuthology.orchestra.run.smithi082.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:42.551 INFO:teuthology.orchestra.run.smithi082.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:42.551 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:42.552 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:42.552 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:42.552 INFO:teuthology.orchestra.run.smithi082.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:42.552 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:d8:da:c6 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:42.552 INFO:teuthology.orchestra.run.smithi082.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:d8:da:c7 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:42.553 INFO:teuthology.orchestra.run.smithi082.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:42.553 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:8f:d3:fe brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:42.553 INFO:teuthology.orchestra.run.smithi082.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-04T18:51:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:8f:d3:ff brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: inet 172.21.15.82/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-04T18:51:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft 42167sec preferred_lft 42167sec 2023-07-04T18:51:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::472f:67ea:567e:8bf7/64 scope link noprefixroute 2023-07-04T18:51:42.554 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:42.555 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-04T18:51:42.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.556 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:42.557 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:42.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link add name ceph-brx type bridge 2023-07-04T18:51:42.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip addr flush dev ceph-brx 2023-07-04T18:51:42.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link set ceph-brx up 2023-07-04T18:51:42.558 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-04T18:51:42.558 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:42.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.724 DEBUG:teuthology.orchestra.run.smithi082:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-04T18:51:42.752 INFO:teuthology.orchestra.run.smithi082.stdout:1 2023-07-04T18:51:42.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.754 DEBUG:teuthology.orchestra.run.smithi082:> route 2023-07-04T18:51:42.823 INFO:teuthology.orchestra.run.smithi082.stdout:Kernel IP routing table 2023-07-04T18:51:42.823 INFO:teuthology.orchestra.run.smithi082.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-04T18:51:42.824 INFO:teuthology.orchestra.run.smithi082.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-04T18:51:42.824 INFO:teuthology.orchestra.run.smithi082.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-04T18:51:42.824 INFO:teuthology.orchestra.run.smithi082.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-04T18:51:42.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:42.825 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:42.826 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:42.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-04T18:51:42.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-04T18:51:42.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-04T18:51:42.828 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:43.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.095 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list 2023-07-04T18:51:43.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.113 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list-id 2023-07-04T18:51:43.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.169 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:43.169 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:43.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-07-04T18:51:43.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-07-04T18:51:43.170 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:43.263 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-07-04T18:51:43.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.264 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:43.265 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:43.265 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-07-04T18:51:43.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-04T18:51:43.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-07-04T18:51:43.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-07-04T18:51:43.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-07-04T18:51:43.268 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:43.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.498 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:43.499 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:43.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link set brx.0 up 2023-07-04T18:51:43.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link set dev brx.0 master ceph-brx 2023-07-04T18:51:43.500 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:43.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:43.608 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-07-04T18:51:43.631 INFO:teuthology.orchestra.run.smithi082.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-07-04T18:51:43.631 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:43.632 DEBUG:teuthology.orchestra.run.smithi082:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-07-04T18:51:43.688 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-07-04T18:51:43.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:43.780 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:43.781 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:43.903 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:43.903 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:43.903 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:43.904 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with new device syntax: 1@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:43.904 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:43.904 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:43.904 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:43.904 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:43.935 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:43.935 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:43.935 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with new device syntax: 1@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790" 2023-07-04T18:51:43.936 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790:/ 2023-07-04T18:51:43.937 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5" will pass to kernel 2023-07-04T18:51:43.953 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-04T18:51:43.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-07-04T18:51:43.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:43.983 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:44.054 INFO:teuthology.orchestra.run.smithi082.stdout:client4704 2023-07-04T18:51:44.055 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:51:44.055 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:44.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.111 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:44.183 INFO:teuthology.orchestra.run.smithi082.stdout:client4704 2023-07-04T18:51:44.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.185 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:44.258 INFO:teuthology.orchestra.run.smithi082.stdout:client4704 2023-07-04T18:51:44.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.259 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.client4704/status') 2023-07-04T18:51:44.339 INFO:teuthology.orchestra.run.smithi082.stdout:instance: client.4704 (3)172.21.15.82:0/2467978996 2023-07-04T18:51:44.339 INFO:teuthology.orchestra.run.smithi082.stdout:blocklisted: false 2023-07-04T18:51:44.340 INFO:teuthology.orchestra.run.smithi082.stderr:0+1 records in 2023-07-04T18:51:44.340 INFO:teuthology.orchestra.run.smithi082.stderr:0+1 records out 2023-07-04T18:51:44.341 INFO:teuthology.orchestra.run.smithi082.stderr:70 bytes copied, 3.7287e-05 s, 1.9 MB/s 2023-07-04T18:51:44.341 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-07-04T18:51:44.341 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-04T18:51:44.342 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-07-04T18:51:44.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-07-04T18:51:44.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.474+0000 7f45c1946700 1 Processor -- start 2023-07-04T18:51:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.474+0000 7f45c1946700 1 -- start start 2023-07-04T18:51:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45c1946700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45bc06a550 0x7f45bc0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45c1946700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc0cb6d0 con 0x7f45bc0716a0 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45c1946700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc0af2e0 con 0x7f45bc0aef10 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45c1946700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc070d80 con 0x7f45bc06a550 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45ba7fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45bc06a550 0x7f45bc0cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45bb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f45bc0716a0 0x7f45bc0cc0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51860/0 (socket says 172.21.15.77:51860) 2023-07-04T18:51:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.475+0000 7f45bb7fe700 1 -- 172.21.15.77:0/1228981591 learned_addr learned my addr 172.21.15.77:0/1228981591 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:44.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1228981591 >> v1:172.21.15.82:6789/0 conn(0x7f45bc0aef10 legacy=0x7f45bc069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:44.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1228981591 >> v1:172.21.15.77:6789/0 conn(0x7f45bc0716a0 legacy=0x7f45bc0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1228981591 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0cb080 con 0x7f45bc06a550 2023-07-04T18:51:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45b9ffb700 1 -- 172.21.15.77:0/1228981591 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702760601 0 0) 0x7f45bc0cb6d0 con 0x7f45bc0716a0 2023-07-04T18:51:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45ba7fc700 1 --2- 172.21.15.77:0/1228981591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc06a550 0x7f45bc0cbb60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009840 tx=0x7f45ac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e820d9e4e902eb70 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45b9ffb700 1 -- 172.21.15.77:0/1228981591 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac004230 con 0x7f45bc06a550 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45b9ffb700 1 -- 172.21.15.77:0/1228981591 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac004390 con 0x7f45bc06a550 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.476+0000 7f45b9ffb700 1 -- 172.21.15.77:0/1228981591 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0036a0 con 0x7f45bc06a550 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 -- 172.21.15.77:0/1228981591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc06a550 msgr2=0x7f45bc0cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 --2- 172.21.15.77:0/1228981591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc06a550 0x7f45bc0cbb60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009840 tx=0x7f45ac009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 -- 172.21.15.77:0/1228981591 shutdown_connections 2023-07-04T18:51:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 --2- 172.21.15.77:0/1228981591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc06a550 0x7f45bc0cbb60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 -- 172.21.15.77:0/1228981591 >> 172.21.15.77:0/1228981591 conn(0x7f45bc06bba0 msgr2=0x7f45bc0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 -- 172.21.15.77:0/1228981591 shutdown_connections 2023-07-04T18:51:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.477+0000 7f45c1946700 1 -- 172.21.15.77:0/1228981591 wait complete. 2023-07-04T18:51:44.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.478+0000 7f45c1946700 1 Processor -- start 2023-07-04T18:51:44.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.478+0000 7f45c1946700 1 -- start start 2023-07-04T18:51:44.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45bc06a550 0x7f45bc1ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45bc0aef10 0x7f45bc15c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc1efb40 con 0x7f45bc0716a0 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0b3090 con 0x7f45bc0aef10 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45c1946700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0b3370 con 0x7f45bc06a550 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45ba7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:44.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45ba7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44726/0 (socket says 172.21.15.77:44726) 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.479+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1134105235 learned_addr learned my addr 172.21.15.77:0/1134105235 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45baffd700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45bc06a550 0x7f45bc1ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45bb7fe700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45bc0aef10 0x7f45bc15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45bc06a550 msgr2=0x7f45bc1ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45bc06a550 0x7f45bc1ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45bc0aef10 msgr2=0x7f45bc15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45bc0aef10 0x7f45bc15c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 -- 172.21.15.77:0/1134105235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0094a0 con 0x7f45bc0716a0 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45ba7fc700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0090b0 tx=0x7f45ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011630 con 0x7f45bc0716a0 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac011790 con 0x7f45bc0716a0 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc0b3650 con 0x7f45bc0716a0 2023-07-04T18:51:44.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011b40 con 0x7f45bc0716a0 2023-07-04T18:51:44.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.480+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc0b3c10 con 0x7f45bc0716a0 2023-07-04T18:51:44.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.481+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc04e210 con 0x7f45bc0716a0 2023-07-04T18:51:44.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.481+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f45ac011ca0 con 0x7f45bc0716a0 2023-07-04T18:51:44.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.482+0000 7f45a3fff700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 0x7f45a806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:44.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.482+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f45ac08ae10 con 0x7f45bc0716a0 2023-07-04T18:51:44.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.482+0000 7f45baffd700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 0x7f45a806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:44.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.482+0000 7f45baffd700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 0x7f45a806d1c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f45b0004f30 tx=0x7f45b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:44.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.486+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f45ac08e030 con 0x7f45bc0716a0 2023-07-04T18:51:44.718 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.716+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f45bc0b2070 con 0x7f45bc0716a0 2023-07-04T18:51:44.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.716+0000 7f45a3fff700 1 -- 172.21.15.77:0/1134105235 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f45ac055eb0 con 0x7f45bc0716a0 2023-07-04T18:51:44.719 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-04T18:51:44.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 msgr2=0x7f45a806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 0x7f45a806d1c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f45b0004f30 tx=0x7f45b0009040 comp rx=0 tx=0).stop 2023-07-04T18:51:44.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 msgr2=0x7f45bc15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:44.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0090b0 tx=0x7f45ac0047c0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.723 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 shutdown_connections 2023-07-04T18:51:44.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f45a806ad10 0x7f45a806d1c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45bc0aef10 0x7f45bc15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45bc06a550 0x7f45bc1ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 --2- 172.21.15.77:0/1134105235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45bc0716a0 0x7f45bc15a8b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:44.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 >> 172.21.15.77:0/1134105235 conn(0x7f45bc06bba0 msgr2=0x7f45bc0b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 shutdown_connections 2023-07-04T18:51:44.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:44.718+0000 7f45c1946700 1 -- 172.21.15.77:0/1134105235 wait complete. 2023-07-04T18:51:44.732 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-04T18:51:44.732 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-04T18:51:44.732 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi082.front.sepia.ceph.com 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-04T18:51:44.733 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-07-04T18:51:44.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.734 DEBUG:teuthology.orchestra.run.smithi082:> ip addr 2023-07-04T18:51:44.752 INFO:teuthology.orchestra.run.smithi082.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:44.752 INFO:teuthology.orchestra.run.smithi082.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:44.752 INFO:teuthology.orchestra.run.smithi082.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:44.752 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.752 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:d8:da:c6 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:d8:da:c7 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:51:44.753 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:8f:d3:fe brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 0c:c4:7a:8f:d3:ff brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout: inet 172.21.15.82/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft 42164sec preferred_lft 42164sec 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::472f:67ea:567e:8bf7/64 scope link noprefixroute 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.754 INFO:teuthology.orchestra.run.smithi082.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 66:f9:5f:8d:a6:96 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::8c3c:a8ff:fe95:46d9/64 scope link 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 66:f9:5f:8d:a6:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-07-04T18:51:44.755 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::64f9:5fff:fe8d:a696/64 scope link tentative 2023-07-04T18:51:44.756 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:44.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.756 DEBUG:teuthology.orchestra.run.smithi082:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-04T18:51:44.817 INFO:teuthology.orchestra.run.smithi082.stdout:1 2023-07-04T18:51:44.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.819 DEBUG:teuthology.orchestra.run.smithi082:> route 2023-07-04T18:51:44.878 INFO:teuthology.orchestra.run.smithi082.stdout:Kernel IP routing table 2023-07-04T18:51:44.879 INFO:teuthology.orchestra.run.smithi082.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-04T18:51:44.879 INFO:teuthology.orchestra.run.smithi082.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-04T18:51:44.879 INFO:teuthology.orchestra.run.smithi082.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-04T18:51:44.879 INFO:teuthology.orchestra.run.smithi082.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-04T18:51:44.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.880 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:44.880 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:44.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-04T18:51:44.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-04T18:51:44.881 DEBUG:teuthology.orchestra.run.smithi082:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-04T18:51:44.881 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:44.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:44.984 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list 2023-07-04T18:51:45.002 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-07-04T18:51:45.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.004 DEBUG:teuthology.orchestra.run.smithi082:> ip netns list-id 2023-07-04T18:51:45.059 INFO:teuthology.orchestra.run.smithi082.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-07-04T18:51:45.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.060 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:45.060 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:45.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-07-04T18:51:45.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-07-04T18:51:45.061 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:45.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.155 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-07-04T18:51:45.194 INFO:teuthology.orchestra.run.smithi082.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:45.194 INFO:teuthology.orchestra.run.smithi082.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:45.194 INFO:teuthology.orchestra.run.smithi082.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:45.195 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.195 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:45.195 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.195 INFO:teuthology.orchestra.run.smithi082.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:45.196 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 4a:38:9a:c2:aa:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-07-04T18:51:45.196 INFO:teuthology.orchestra.run.smithi082.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-07-04T18:51:45.196 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.196 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::4838:9aff:fec2:aa7c/64 scope link 2023-07-04T18:51:45.196 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.207 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-07-04T18:51:45.288 INFO:teuthology.orchestra.run.smithi082.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:51:45.288 INFO:teuthology.orchestra.run.smithi082.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 ::1/128 scope host 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:51:45.289 INFO:teuthology.orchestra.run.smithi082.stdout: link/ether 4a:38:9a:c2:aa:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-07-04T18:51:45.290 INFO:teuthology.orchestra.run.smithi082.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-07-04T18:51:45.290 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.290 INFO:teuthology.orchestra.run.smithi082.stdout: inet6 fe80::4838:9aff:fec2:aa7c/64 scope link 2023-07-04T18:51:45.290 INFO:teuthology.orchestra.run.smithi082.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:51:45.314 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-07-04T18:51:45.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.321 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:45.321 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:45.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-07-04T18:51:45.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-07-04T18:51:45.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-07-04T18:51:45.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-07-04T18:51:45.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-07-04T18:51:45.323 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:45.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.552 DEBUG:teuthology.orchestra.run.smithi082:> (cd / && exec bash -c ' 2023-07-04T18:51:45.552 DEBUG:teuthology.orchestra.run.smithi082:> set -e 2023-07-04T18:51:45.552 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link set brx.1 up 2023-07-04T18:51:45.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo ip link set dev brx.1 master ceph-brx 2023-07-04T18:51:45.553 DEBUG:teuthology.orchestra.run.smithi082:> ') 2023-07-04T18:51:45.658 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:45.659 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-07-04T18:51:45.696 INFO:teuthology.orchestra.run.smithi082.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-07-04T18:51:45.697 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:51:45.697 DEBUG:teuthology.orchestra.run.smithi082:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-07-04T18:51:45.761 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-07-04T18:51:45.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.762 DEBUG:teuthology.orchestra.run.smithi082:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:45.846 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:45.846 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:45.871 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:45.871 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:45.871 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:45.872 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with new device syntax: 3@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:45.872 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:45.872 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:45.872 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:45.873 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:45.896 INFO:teuthology.orchestra.run.smithi082.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with new device syntax: 3@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:51:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:51:45.898 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:51:45.898 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790" 2023-07-04T18:51:45.898 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790:/ 2023-07-04T18:51:45.898 INFO:teuthology.orchestra.run.smithi082.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,key=3,mds_namespace=cephfs,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5" will pass to kernel 2023-07-04T18:51:45.900 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-04T18:51:45.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-07-04T18:51:45.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:45.931 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:46.003 INFO:teuthology.orchestra.run.smithi082.stdout:client4600 2023-07-04T18:51:46.003 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:51:46.004 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:46.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:46.060 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:46.135 INFO:teuthology.orchestra.run.smithi082.stdout:client4600 2023-07-04T18:51:46.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:46.137 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:51:46.207 INFO:teuthology.orchestra.run.smithi082.stdout:client4600 2023-07-04T18:51:46.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:46.208 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.client4600/status') 2023-07-04T18:51:46.287 INFO:teuthology.orchestra.run.smithi082.stdout:instance: client.4600 (3)192.168.0.2:0/1210166835 2023-07-04T18:51:46.288 INFO:teuthology.orchestra.run.smithi082.stdout:blocklisted: false 2023-07-04T18:51:46.289 INFO:teuthology.orchestra.run.smithi082.stderr:0+1 records in 2023-07-04T18:51:46.289 INFO:teuthology.orchestra.run.smithi082.stderr:0+1 records out 2023-07-04T18:51:46.289 INFO:teuthology.orchestra.run.smithi082.stderr:69 bytes copied, 3.603e-05 s, 1.9 MB/s 2023-07-04T18:51:46.289 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-07-04T18:51:46.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:46.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.445+0000 7f031b67e700 1 Processor -- start 2023-07-04T18:51:46.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.445+0000 7f031b67e700 1 -- start start 2023-07-04T18:51:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f031b67e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 0x7f03141641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f031b67e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031406d890 con 0x7f031406dd20 2023-07-04T18:51:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f031b67e700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031406cc80 con 0x7f0314159280 2023-07-04T18:51:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f031b67e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031406cf60 con 0x7f03140cdca0 2023-07-04T18:51:46.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0318c19700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f031406dd20 0x7f03140cd590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51878/0 (socket says 172.21.15.77:51878) 2023-07-04T18:51:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0318c19700 1 -- 172.21.15.77:0/4099537297 learned_addr learned my addr 172.21.15.77:0/4099537297 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0319c1b700 1 --2- 172.21.15.77:0/4099537297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 0x7f03141641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777030423 0 0) 0x7f031406d890 con 0x7f031406dd20 2023-07-04T18:51:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000035b0 con 0x7f031406dd20 2023-07-04T18:51:46.451 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499370878 0 0) 0x7f031406cc80 con 0x7f0314159280 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031406d890 con 0x7f0314159280 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102648163 0 0) 0x7f03000035b0 con 0x7f031406dd20 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0319c1b700 1 -- 172.21.15.77:0/4099537297 >> v1:172.21.15.82:6789/0 conn(0x7f0314159280 legacy=0x7f031406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0319c1b700 1 -- 172.21.15.77:0/4099537297 >> v1:172.21.15.77:6789/0 conn(0x7f031406dd20 legacy=0x7f03140cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0319c1b700 1 -- 172.21.15.77:0/4099537297 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031406d240 con 0x7f03140cdca0 2023-07-04T18:51:46.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.446+0000 7f0319c1b700 1 --2- 172.21.15.77:0/4099537297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 0x7f03141641e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f030c009dc0 tx=0x7f030c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e0310bdbe74715c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c00e070 con 0x7f03140cdca0 2023-07-04T18:51:46.453 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c003a20 con 0x7f03140cdca0 2023-07-04T18:51:46.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f030bfff700 1 -- 172.21.15.77:0/4099537297 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0310001670 con 0x7f03140cdca0 2023-07-04T18:51:46.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 -- 172.21.15.77:0/4099537297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 msgr2=0x7f03141641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.455 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 --2- 172.21.15.77:0/4099537297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 0x7f03141641e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f030c009dc0 tx=0x7f030c00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:46.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 -- 172.21.15.77:0/4099537297 shutdown_connections 2023-07-04T18:51:46.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 --2- 172.21.15.77:0/4099537297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f03140cdca0 0x7f03141641e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 -- 172.21.15.77:0/4099537297 >> 172.21.15.77:0/4099537297 conn(0x7f031406bba0 msgr2=0x7f03140b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:46.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 -- 172.21.15.77:0/4099537297 shutdown_connections 2023-07-04T18:51:46.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.447+0000 7f031b67e700 1 -- 172.21.15.77:0/4099537297 wait complete. 2023-07-04T18:51:46.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.448+0000 7f031b67e700 1 Processor -- start 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.448+0000 7f031b67e700 1 -- start start 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 0x7f03140c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0314159280 0x7f03140c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140c3690 con 0x7f031406dd20 2023-07-04T18:51:46.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140c3970 con 0x7f03140cdca0 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031b67e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03140c3c50 con 0x7f0314159280 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031941a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031941a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44746/0 (socket says 172.21.15.77:44746) 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f031941a700 1 -- 172.21.15.77:0/3471640736 learned_addr learned my addr 172.21.15.77:0/3471640736 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f0319c1b700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0314159280 0x7f03140c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.449+0000 7f0318c19700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 0x7f03140c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 -- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0314159280 msgr2=0x7f03140c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.461 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0314159280 0x7f03140c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 -- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 msgr2=0x7f03140c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 0x7f03140c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 -- 172.21.15.77:0/3471640736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c009a70 con 0x7f031406dd20 2023-07-04T18:51:46.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f0318c19700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 0x7f03140c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:46.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031941a700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f030400c520 tx=0x7f030400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03040052d0 con 0x7f031406dd20 2023-07-04T18:51:46.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304005430 con 0x7f031406dd20 2023-07-04T18:51:46.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03140c3f30 con 0x7f031406dd20 2023-07-04T18:51:46.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03040038a0 con 0x7f031406dd20 2023-07-04T18:51:46.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.450+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03141efb60 con 0x7f031406dd20 2023-07-04T18:51:46.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.451+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031404e210 con 0x7f031406dd20 2023-07-04T18:51:46.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.451+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0304003a00 con 0x7f031406dd20 2023-07-04T18:51:46.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.452+0000 7f0309ffb700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 0x7f02f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.452+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0304088a30 con 0x7f031406dd20 2023-07-04T18:51:46.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.452+0000 7f0318c19700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 0x7f02f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.452+0000 7f0318c19700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 0x7f02f806d190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03100017a0 tx=0x7f0310001550 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.456+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0304053a60 con 0x7f031406dd20 2023-07-04T18:51:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.645+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03140c37d0 con 0x7f031406dd20 2023-07-04T18:51:46.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.647+0000 7f0309ffb700 1 -- 172.21.15.77:0/3471640736 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f03140c37d0 con 0x7f031406dd20 2023-07-04T18:51:46.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:46.651 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:46.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 msgr2=0x7f02f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 0x7f02f806d190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03100017a0 tx=0x7f0310001550 comp rx=0 tx=0).stop 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 msgr2=0x7f03141ef440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f030400c520 tx=0x7f030400c830 comp rx=0 tx=0).stop 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 shutdown_connections 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f02f806ace0 0x7f02f806d190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f031406dd20 0x7f03141ef440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f0314159280 0x7f03140c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 --2- 172.21.15.77:0/3471640736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03140cdca0 0x7f03140c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 >> 172.21.15.77:0/3471640736 conn(0x7f031406bba0 msgr2=0x7f03140b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 shutdown_connections 2023-07-04T18:51:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.649+0000 7f031b67e700 1 -- 172.21.15.77:0/3471640736 wait complete. 2023-07-04T18:51:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:46.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:46.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.789+0000 7f18b435a700 1 Processor -- start 2023-07-04T18:51:46.791 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.789+0000 7f18b435a700 1 -- start start 2023-07-04T18:51:46.792 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b435a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac069e40 0x7f18ac06a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b435a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18ac0aeb40 0x7f18ac15db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b435a700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0cf750 con 0x7f18ac0d01c0 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b435a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ac0af060 con 0x7f18ac0aeb40 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b435a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac0cf9c0 con 0x7f18ac069e40 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b28f7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18ac0aeb40 0x7f18ac15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.793 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.790+0000 7f18b28f7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f18ac0aeb40 0x7f18ac15db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44758/0 (socket says 172.21.15.77:44758) 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 -- 172.21.15.77:0/2449475005 learned_addr learned my addr 172.21.15.77:0/2449475005 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b20f6700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac069e40 0x7f18ac06a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 -- 172.21.15.77:0/2449475005 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac069e40 msgr2=0x7f18ac06a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac069e40 0x7f18ac06a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 -- 172.21.15.77:0/2449475005 >> v1:172.21.15.82:6789/0 conn(0x7f18ac0d01c0 legacy=0x7f18ac0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 -- 172.21.15.77:0/2449475005 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0cfca0 con 0x7f18ac0aeb40 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b28f7700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0aeb40 0x7f18ac15db70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009e50 tx=0x7f18a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e886f614499967 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.794 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b10f4700 1 -- 172.21.15.77:0/2449475005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a400e070 con 0x7f18ac0aeb40 2023-07-04T18:51:46.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b10f4700 1 -- 172.21.15.77:0/2449475005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a4003a20 con 0x7f18ac0aeb40 2023-07-04T18:51:46.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.791+0000 7f18b10f4700 1 -- 172.21.15.77:0/2449475005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4003dd0 con 0x7f18ac0aeb40 2023-07-04T18:51:46.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 -- 172.21.15.77:0/2449475005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0aeb40 msgr2=0x7f18ac15db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0aeb40 0x7f18ac15db70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009e50 tx=0x7f18a400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:46.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 -- 172.21.15.77:0/2449475005 shutdown_connections 2023-07-04T18:51:46.797 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac0aeb40 0x7f18ac15db70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 --2- 172.21.15.77:0/2449475005 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac069e40 0x7f18ac06a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 -- 172.21.15.77:0/2449475005 >> 172.21.15.77:0/2449475005 conn(0x7f18ac06af50 msgr2=0x7f18ac06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 -- 172.21.15.77:0/2449475005 shutdown_connections 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.792+0000 7f18b435a700 1 -- 172.21.15.77:0/2449475005 wait complete. 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.793+0000 7f18b435a700 1 Processor -- start 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.793+0000 7f18b435a700 1 -- start start 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac0aeb40 0x7f18ac15f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f18ac0d01c0 0x7f18ac1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15f9a0 con 0x7f18ac069e40 2023-07-04T18:51:46.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15c3e0 con 0x7f18ac0d01c0 2023-07-04T18:51:46.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac15f6c0 con 0x7f18ac0aeb40 2023-07-04T18:51:46.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44760/0 (socket says 172.21.15.77:44760) 2023-07-04T18:51:46.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 -- 172.21.15.77:0/107475293 learned_addr learned my addr 172.21.15.77:0/107475293 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b18f5700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac0aeb40 0x7f18ac15f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b28f7700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f18ac0d01c0 0x7f18ac1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 -- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac0aeb40 msgr2=0x7f18ac15f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac0aeb40 0x7f18ac15f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 -- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f18ac0d01c0 msgr2=0x7f18ac1413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f18ac0d01c0 0x7f18ac1413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 -- 172.21.15.77:0/107475293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009b30 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b20f6700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c7f0 tx=0x7f189c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c011800 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c011960 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac1f1bf0 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c011cd0 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.794+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac1f21b0 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.795+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac04e210 con 0x7f18ac069e40 2023-07-04T18:51:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.796+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f189c004020 con 0x7f18ac069e40 2023-07-04T18:51:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.796+0000 7f18a2ffd700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 0x7f189006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.796+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f189c088f60 con 0x7f18ac069e40 2023-07-04T18:51:46.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.796+0000 7f18b18f5700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 0x7f189006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:46.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.797+0000 7f18b18f5700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 0x7f189006d190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f18a8007fd0 tx=0x7f18a8007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:46.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.800+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f189c054010 con 0x7f18ac069e40 2023-07-04T18:51:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.984+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18ac071570 con 0x7f18ac069e40 2023-07-04T18:51:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.984+0000 7f18a2ffd700 1 -- 172.21.15.77:0/107475293 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f189c057630 con 0x7f18ac069e40 2023-07-04T18:51:46.987 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 msgr2=0x7f189006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 0x7f189006d190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f18a8007fd0 tx=0x7f18a8007ec0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 msgr2=0x7f18ac15c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f189c00c7f0 tx=0x7f189c00cb00 comp rx=0 tx=0).stop 2023-07-04T18:51:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 shutdown_connections 2023-07-04T18:51:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f189006ace0 0x7f189006d190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18ac069e40 0x7f18ac15c910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f18ac0d01c0 0x7f18ac1413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 --2- 172.21.15.77:0/107475293 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f18ac0aeb40 0x7f18ac15f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:46.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 >> 172.21.15.77:0/107475293 conn(0x7f18ac06af50 msgr2=0x7f18ac145a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:46.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 shutdown_connections 2023-07-04T18:51:46.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:46.986+0000 7f18b435a700 1 -- 172.21.15.77:0/107475293 wait complete. 2023-07-04T18:51:47.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.121+0000 7f5965129700 1 Processor -- start 2023-07-04T18:51:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.121+0000 7f5965129700 1 -- start start 2023-07-04T18:51:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f5965129700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f596006d5d0 0x7f59600aded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f5965129700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59600ca8a0 con 0x7f59600708a0 2023-07-04T18:51:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f5965129700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59600cb3b0 con 0x7f59600adb00 2023-07-04T18:51:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f5965129700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59600cce60 con 0x7f596006d5d0 2023-07-04T18:51:47.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595f59e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f59600708a0 0x7f59600cd2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51910/0 (socket says 172.21.15.77:51910) 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595f59e700 1 -- 172.21.15.77:0/3605067754 learned_addr learned my addr 172.21.15.77:0/3605067754 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595e59c700 1 --2- 172.21.15.77:0/3605067754 >> v2:172.21.15.82:3300/0 conn(0x7f596006d5d0 0x7f59600aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162215332 0 0) 0x7f59600ca8a0 con 0x7f59600708a0 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59480035b0 con 0x7f59600708a0 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117449449 0 0) 0x7f59600cb3b0 con 0x7f59600adb00 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.122+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59600ca8a0 con 0x7f59600adb00 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595e59c700 1 -- 172.21.15.77:0/3605067754 >> v1:172.21.15.82:6789/0 conn(0x7f59600adb00 legacy=0x7f59600ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595e59c700 1 -- 172.21.15.77:0/3605067754 >> v1:172.21.15.77:6789/0 conn(0x7f59600708a0 legacy=0x7f59600cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595e59c700 1 -- 172.21.15.77:0/3605067754 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59600cab10 con 0x7f596006d5d0 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595e59c700 1 --2- 172.21.15.77:0/3605067754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596006d5d0 0x7f59600aded0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f595000a9e0 tx=0x7f595000acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f89bb0498b6854 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595000cb70 con 0x7f596006d5d0 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595000ccd0 con 0x7f596006d5d0 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f595dd9b700 1 -- 172.21.15.77:0/3605067754 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950018620 con 0x7f596006d5d0 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 -- 172.21.15.77:0/3605067754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596006d5d0 msgr2=0x7f59600aded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 --2- 172.21.15.77:0/3605067754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596006d5d0 0x7f59600aded0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f595000a9e0 tx=0x7f595000acf0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 -- 172.21.15.77:0/3605067754 shutdown_connections 2023-07-04T18:51:47.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 --2- 172.21.15.77:0/3605067754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596006d5d0 0x7f59600aded0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 -- 172.21.15.77:0/3605067754 >> 172.21.15.77:0/3605067754 conn(0x7f59600cf0d0 msgr2=0x7f59600cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:47.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 -- 172.21.15.77:0/3605067754 shutdown_connections 2023-07-04T18:51:47.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.123+0000 7f5965129700 1 -- 172.21.15.77:0/3605067754 wait complete. 2023-07-04T18:51:47.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.125+0000 7f5965129700 1 Processor -- start 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.125+0000 7f5965129700 1 -- start start 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59600708a0 0x7f59600c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f59600adb00 0x7f596006c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600c6890 con 0x7f596006d5d0 2023-07-04T18:51:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596009b900 con 0x7f59600708a0 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596009bbe0 con 0x7f59600adb00 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44774/0 (socket says 172.21.15.77:44774) 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 -- 172.21.15.77:0/1088468264 learned_addr learned my addr 172.21.15.77:0/1088468264 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595e59c700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59600708a0 0x7f59600c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595f59e700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f59600adb00 0x7f596006c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 -- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f59600adb00 msgr2=0x7f596006c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f59600adb00 0x7f596006c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 -- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59600708a0 msgr2=0x7f59600c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59600708a0 0x7f59600c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 -- 172.21.15.77:0/1088468264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595000a640 con 0x7f596006d5d0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f595ed9d700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f595400c520 tx=0x7f595400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540052d0 con 0x7f596006d5d0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954005430 con 0x7f596006d5d0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59600c6240 con 0x7f596006d5d0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.126+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540038c0 con 0x7f596006d5d0 2023-07-04T18:51:47.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.127+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59601582a0 con 0x7f596006d5d0 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.127+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596004e210 con 0x7f596006d5d0 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.128+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5954003a20 con 0x7f596006d5d0 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.128+0000 7f59477fe700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 0x7f594c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.128+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5954088ac0 con 0x7f596006d5d0 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.128+0000 7f595e59c700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 0x7f594c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.129+0000 7f595e59c700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 0x7f594c06d190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f595000a1c0 tx=0x7f595001a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.133+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5954053af0 con 0x7f596006d5d0 2023-07-04T18:51:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.324+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f59600b3bf0 con 0x7f596006d5d0 2023-07-04T18:51:47.325 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.324+0000 7f59477fe700 1 -- 172.21.15.77:0/1088468264 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5954057110 con 0x7f596006d5d0 2023-07-04T18:51:47.326 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:47.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 msgr2=0x7f594c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 0x7f594c06d190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f595000a1c0 tx=0x7f595001a040 comp rx=0 tx=0).stop 2023-07-04T18:51:47.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 msgr2=0x7f5960157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f595400c520 tx=0x7f595400c8e0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 shutdown_connections 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f594c06ace0 0x7f594c06d190 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f596006d5d0 0x7f5960157d60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f59600adb00 0x7f596006c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 --2- 172.21.15.77:0/1088468264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59600708a0 0x7f59600c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 >> 172.21.15.77:0/1088468264 conn(0x7f59600cf0d0 msgr2=0x7f5960158620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 shutdown_connections 2023-07-04T18:51:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.326+0000 7f5965129700 1 -- 172.21.15.77:0/1088468264 wait complete. 2023-07-04T18:51:47.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:47.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.463+0000 7f2cad4d8700 1 Processor -- start 2023-07-04T18:51:47.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.463+0000 7f2cad4d8700 1 -- start start 2023-07-04T18:51:47.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2cad4d8700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2ca80b3330 0x7f2ca80b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2cad4d8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca80b3f60 con 0x7f2ca806cfc0 2023-07-04T18:51:47.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2cad4d8700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca806c4e0 con 0x7f2ca81590b0 2023-07-04T18:51:47.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2cad4d8700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca806caa0 con 0x7f2ca80b3330 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2ca77fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ca806cfc0 0x7f2ca806d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51924/0 (socket says 172.21.15.77:51924) 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2ca77fe700 1 -- 172.21.15.77:0/2573775027 learned_addr learned my addr 172.21.15.77:0/2573775027 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2ca67fc700 1 --2- 172.21.15.77:0/2573775027 >> v2:172.21.15.82:3300/0 conn(0x7f2ca80b3330 0x7f2ca80b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567378609 0 0) 0x7f2ca80b3f60 con 0x7f2ca806cfc0 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.464+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c900035b0 con 0x7f2ca806cfc0 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424792958 0 0) 0x7f2c900035b0 con 0x7f2ca806cfc0 2023-07-04T18:51:47.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 >> v2:172.21.15.82:3300/0 conn(0x7f2ca80b3330 msgr2=0x7f2ca80b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 --2- 172.21.15.77:0/2573775027 >> v2:172.21.15.82:3300/0 conn(0x7f2ca80b3330 0x7f2ca80b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 >> v1:172.21.15.82:6789/0 conn(0x7f2ca81590b0 legacy=0x7f2ca815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca806c7c0 con 0x7f2ca806cfc0 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f2ca00019f0 con 0x7f2ca806cfc0 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 >> v1:172.21.15.77:6789/0 conn(0x7f2ca806cfc0 legacy=0x7f2ca806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 0x7f2c90003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c90004850 0x7f2c90004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90005200 0x7f2c9000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca806caa0 con 0x7f2c90003a00 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9000c110 con 0x7f2c90005200 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9000c2d0 con 0x7f2c90004850 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 0x7f2c90003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca77fe700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90005200 0x7f2c9000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca67fc700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c90004850 0x7f2c90004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 -- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c90004850 msgr2=0x7f2c90004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c90004850 0x7f2c90004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 -- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90005200 msgr2=0x7f2c9000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.472 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90005200 0x7f2c9000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 -- 172.21.15.77:0/2573775027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c900035b0 con 0x7f2c90003a00 2023-07-04T18:51:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca77fe700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90005200 0x7f2c9000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca77fe700 1 -- 172.21.15.77:0/2573775027 reap_dead start 2023-07-04T18:51:47.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 0x7f2c90003e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0017c0 tx=0x7f2c90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a306de23a3cc29e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c0052d0 con 0x7f2c90003a00 2023-07-04T18:51:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9c005430 con 0x7f2c90003a00 2023-07-04T18:51:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.465+0000 7f2ca5ffb700 1 -- 172.21.15.77:0/2573775027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c0038c0 con 0x7f2c90003a00 2023-07-04T18:51:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 -- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 msgr2=0x7f2c90003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 0x7f2c90003e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0017c0 tx=0x7f2c90005880 comp rx=0 tx=0).stop 2023-07-04T18:51:47.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 -- 172.21.15.77:0/2573775027 shutdown_connections 2023-07-04T18:51:47.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/2573775027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c90003a00 0x7f2c90003e10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 -- 172.21.15.77:0/2573775027 >> 172.21.15.77:0/2573775027 conn(0x7f2ca806af50 msgr2=0x7f2ca806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:47.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 -- 172.21.15.77:0/2573775027 shutdown_connections 2023-07-04T18:51:47.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.466+0000 7f2cad4d8700 1 -- 172.21.15.77:0/2573775027 wait complete. 2023-07-04T18:51:47.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.467+0000 7f2cad4d8700 1 Processor -- start 2023-07-04T18:51:47.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.467+0000 7f2cad4d8700 1 -- start start 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 0x7f2ca814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2ca80af440 0x7f2ca80af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca80afdf0 con 0x7f2ca8155250 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca81eeb90 con 0x7f2c90003a00 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2cad4d8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca814da80 con 0x7f2ca80af440 2023-07-04T18:51:47.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44796/0 (socket says 172.21.15.77:44796) 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 -- 172.21.15.77:0/3245801402 learned_addr learned my addr 172.21.15.77:0/3245801402 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca77fe700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2ca80af440 0x7f2ca80af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 0x7f2ca814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 -- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2ca80af440 msgr2=0x7f2ca80af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2ca80af440 0x7f2ca80af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 -- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 msgr2=0x7f2ca814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 0x7f2ca814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.468+0000 7f2ca67fc700 1 -- 172.21.15.77:0/3245801402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca81559c0 con 0x7f2ca8155250 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 0x7f2ca814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2ca67fc700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c980116f0 tx=0x7f2c98011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98015870 con 0x7f2ca8155250 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c980159d0 con 0x7f2ca8155250 2023-07-04T18:51:47.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca8155ca0 con 0x7f2ca8155250 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98015d40 con 0x7f2ca8155250 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.469+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca8156260 con 0x7f2ca8155250 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.470+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca81eed00 con 0x7f2ca8155250 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.470+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2c98013400 con 0x7f2ca8155250 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.470+0000 7f2c8f7fe700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 0x7f2c9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.471+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f2c98089ba0 con 0x7f2ca8155250 2023-07-04T18:51:47.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.471+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 0x7f2c9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.471+0000 7f2ca6ffd700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 0x7f2c9406d190 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c005040 tx=0x7f2c9c009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.475+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2c98054c50 con 0x7f2ca8155250 2023-07-04T18:51:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.663+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ca804e210 con 0x7f2ca8155250 2023-07-04T18:51:47.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.663+0000 7f2c8f7fe700 1 -- 172.21.15.77:0/3245801402 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c98058270 con 0x7f2ca8155250 2023-07-04T18:51:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 msgr2=0x7f2c9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 0x7f2c9406d190 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c005040 tx=0x7f2c9c009030 comp rx=0 tx=0).stop 2023-07-04T18:51:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 msgr2=0x7f2ca814d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c980116f0 tx=0x7f2c98011ab0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 shutdown_connections 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c9406ace0 0x7f2c9406d190 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2ca80af440 0x7f2ca80af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c90003a00 0x7f2ca814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 --2- 172.21.15.77:0/3245801402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca8155250 0x7f2ca814d540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 >> 172.21.15.77:0/3245801402 conn(0x7f2ca806af50 msgr2=0x7f2ca80b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 shutdown_connections 2023-07-04T18:51:47.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.665+0000 7f2cad4d8700 1 -- 172.21.15.77:0/3245801402 wait complete. 2023-07-04T18:51:47.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:47.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 Processor -- start 2023-07-04T18:51:47.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 -- start start 2023-07-04T18:51:47.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fb20406eb00 0x7fb2040accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb20406a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2040b3e10 con 0x7fb204160940 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.801+0000 7fb20c4d2700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2040b3660 con 0x7fb20406eb00 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20c4d2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20406a7a0 con 0x7fb204069e40 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb209a6d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb204160940 0x7fb2040b2f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51940/0 (socket says 172.21.15.77:51940) 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb209a6d700 1 -- 172.21.15.77:0/1110496625 learned_addr learned my addr 172.21.15.77:0/1110496625 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566775840 0 0) 0x7fb2040b3e10 con 0x7fb204160940 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20a26e700 1 --2- 172.21.15.77:0/1110496625 >> v2:172.21.15.82:3300/0 conn(0x7fb20406eb00 0x7fb2040accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f00035b0 con 0x7fb204160940 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb20406a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597190752 0 0) 0x7fb1f00035b0 con 0x7fb204160940 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 msgr2=0x7fb20406a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb20406a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 >> v2:172.21.15.82:3300/0 conn(0x7fb20406eb00 msgr2=0x7fb2040accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 --2- 172.21.15.77:0/1110496625 >> v2:172.21.15.82:3300/0 conn(0x7fb20406eb00 0x7fb2040accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2040b3850 con 0x7fb204160940 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fb200001670 con 0x7fb204160940 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 >> v1:172.21.15.77:6789/0 conn(0x7fb204160940 legacy=0x7fb2040b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0003a00 0x7fb1f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb1f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f0004d90 0x7fb1f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20406a7a0 con 0x7fb1f0004d90 2023-07-04T18:51:47.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f000ba10 con 0x7fb1f0003a00 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f000bb50 con 0x7fb204069e40 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f0004d90 0x7fb1f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb20a26e700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0003a00 0x7fb1f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.802+0000 7fb209a6d700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb1f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 -- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 msgr2=0x7fb1f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb204069e40 0x7fb1f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 -- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0003a00 msgr2=0x7fb1f0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0003a00 0x7fb1f0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 -- 172.21.15.77:0/1110496625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00035b0 con 0x7fb1f0004d90 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f0004d90 0x7fb1f0005200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009fc0 tx=0x7fb1f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de1677bdb1ebaec server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc00ec30 con 0x7fb1f0004d90 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc00f040 con 0x7fb1f0004d90 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20926c700 1 -- 172.21.15.77:0/1110496625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc014480 con 0x7fb1f0004d90 2023-07-04T18:51:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1110496625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f0004d90 msgr2=0x7fb1f0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1110496625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f0004d90 0x7fb1f0005200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009fc0 tx=0x7fb1f0005880 comp rx=0 tx=0).stop 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.803+0000 7fb20aa6f700 1 -- 172.21.15.77:0/1110496625 reap_dead start 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.804+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1110496625 shutdown_connections 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.804+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1110496625 >> 172.21.15.77:0/1110496625 conn(0x7fb20406af50 msgr2=0x7fb20406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.804+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1110496625 shutdown_connections 2023-07-04T18:51:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.804+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1110496625 wait complete. 2023-07-04T18:51:47.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.805+0000 7fb20c4d2700 1 Processor -- start 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.805+0000 7fb20c4d2700 1 -- start start 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.805+0000 7fb20c4d2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0004d90 0x7fb20406d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.805+0000 7fb20c4d2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb2041e7510 0x7fb204157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204148de0 con 0x7fb20406d780 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20406dbf0 con 0x7fb1f0004d90 2023-07-04T18:51:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041e7d00 con 0x7fb2041e7510 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44812/0 (socket says 172.21.15.77:44812) 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 -- 172.21.15.77:0/1852861830 learned_addr learned my addr 172.21.15.77:0/1852861830 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20aa6f700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb2041e7510 0x7fb204157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20a26e700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0004d90 0x7fb20406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 -- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb2041e7510 msgr2=0x7fb204157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb2041e7510 0x7fb204157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 -- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0004d90 msgr2=0x7fb20406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0004d90 0x7fb20406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 -- 172.21.15.77:0/1852861830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2041e7fe0 con 0x7fb20406d780 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb209a6d700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb200007340 tx=0x7fb200007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000f8d0 con 0x7fb20406d780 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20000fa70 con 0x7fb20406d780 2023-07-04T18:51:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000fe20 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2041e82c0 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.806+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2041e8880 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.807+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20404e210 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.808+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb20001d460 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.808+0000 7fb1faffd700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 0x7fb1e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.808+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fb200089580 con 0x7fb20406d780 2023-07-04T18:51:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.808+0000 7fb20a26e700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 0x7fb1e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:47.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.809+0000 7fb20a26e700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 0x7fb1e806d190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009c20 tx=0x7fb1f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:47.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.812+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb200054630 con 0x7fb20406d780 2023-07-04T18:51:48.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:47.999+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2040cbd40 con 0x7fb20406d780 2023-07-04T18:51:48.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.000+0000 7fb1faffd700 1 -- 172.21.15.77:0/1852861830 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb200057c50 con 0x7fb20406d780 2023-07-04T18:51:48.002 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:48.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 msgr2=0x7fb1e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 0x7fb1e806d190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009c20 tx=0x7fb1f4008040 comp rx=0 tx=0).stop 2023-07-04T18:51:48.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 msgr2=0x7fb2041e6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb200007340 tx=0x7fb200007650 comp rx=0 tx=0).stop 2023-07-04T18:51:48.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 shutdown_connections 2023-07-04T18:51:48.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fb1e806ace0 0x7fb1e806d190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fb2041e7510 0x7fb204157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1f0004d90 0x7fb20406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 --2- 172.21.15.77:0/1852861830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb20406d780 0x7fb2041e6fd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 >> 172.21.15.77:0/1852861830 conn(0x7fb20406af50 msgr2=0x7fb2040b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 shutdown_connections 2023-07-04T18:51:48.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.002+0000 7fb20c4d2700 1 -- 172.21.15.77:0/1852861830 wait complete. 2023-07-04T18:51:48.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.136+0000 7f22a8cba700 1 Processor -- start 2023-07-04T18:51:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.136+0000 7f22a8cba700 1 -- start start 2023-07-04T18:51:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a8cba700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a4159280 0x7f22a406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a8cba700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 0x7f22a40cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a8cba700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a406d890 con 0x7f22a406dd20 2023-07-04T18:51:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a8cba700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a406cc80 con 0x7f22a4159280 2023-07-04T18:51:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a8cba700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a406cf60 con 0x7f22a406c770 2023-07-04T18:51:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a4159280 0x7f22a406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f22a4159280 0x7f22a406c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44820/0 (socket says 172.21.15.77:44820) 2023-07-04T18:51:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 -- 172.21.15.77:0/1791350703 learned_addr learned my addr 172.21.15.77:0/1791350703 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a1d9b700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 0x7f22a40cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 -- 172.21.15.77:0/1791350703 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 msgr2=0x7f22a40cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 0x7f22a40cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 -- 172.21.15.77:0/1791350703 >> v1:172.21.15.82:6789/0 conn(0x7f22a406dd20 legacy=0x7f22a4070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 -- 172.21.15.77:0/1791350703 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a406d240 con 0x7f22a4159280 2023-07-04T18:51:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a1d9b700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 0x7f22a40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:48.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a259c700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a4159280 0x7f22a406c230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f229400bb90 tx=0x7f229400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9581d39526ceccf8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a159a700 1 -- 172.21.15.77:0/1791350703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229400dd10 con 0x7f22a4159280 2023-07-04T18:51:48.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.137+0000 7f22a159a700 1 -- 172.21.15.77:0/1791350703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229400de70 con 0x7f22a4159280 2023-07-04T18:51:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a159a700 1 -- 172.21.15.77:0/1791350703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2294004230 con 0x7f22a4159280 2023-07-04T18:51:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 -- 172.21.15.77:0/1791350703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a4159280 msgr2=0x7f22a406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a4159280 0x7f22a406c230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f229400bb90 tx=0x7f229400bea0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 -- 172.21.15.77:0/1791350703 shutdown_connections 2023-07-04T18:51:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a4159280 0x7f22a406c230 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 --2- 172.21.15.77:0/1791350703 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a406c770 0x7f22a40cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 -- 172.21.15.77:0/1791350703 >> 172.21.15.77:0/1791350703 conn(0x7f22a406bba0 msgr2=0x7f22a40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 -- 172.21.15.77:0/1791350703 shutdown_connections 2023-07-04T18:51:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.138+0000 7f22a8cba700 1 -- 172.21.15.77:0/1791350703 wait complete. 2023-07-04T18:51:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.139+0000 7f22a8cba700 1 Processor -- start 2023-07-04T18:51:48.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 -- start start 2023-07-04T18:51:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22a406c770 0x7f22a414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.152 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a4159280 0x7f22a41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a414b220 con 0x7f22a406dd20 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40c90f0 con 0x7f22a406c770 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a8cba700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40c93d0 con 0x7f22a4159280 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44826/0 (socket says 172.21.15.77:44826) 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.140+0000 7f22a1d9b700 1 -- 172.21.15.77:0/749491032 learned_addr learned my addr 172.21.15.77:0/749491032 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:48.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a2d9d700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a4159280 0x7f22a41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a259c700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22a406c770 0x7f22a414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 -- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a4159280 msgr2=0x7f22a41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a4159280 0x7f22a41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 -- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22a406c770 msgr2=0x7f22a414ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22a406c770 0x7f22a414ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 -- 172.21.15.77:0/749491032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229400b7f0 con 0x7f22a406dd20 2023-07-04T18:51:48.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a1d9b700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f229800c780 tx=0x7f229800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011800 con 0x7f22a406dd20 2023-07-04T18:51:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298011960 con 0x7f22a406dd20 2023-07-04T18:51:48.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a40c96b0 con 0x7f22a406dd20 2023-07-04T18:51:48.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298011d10 con 0x7f22a406dd20 2023-07-04T18:51:48.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.141+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a415c5b0 con 0x7f22a406dd20 2023-07-04T18:51:48.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.142+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a4159e90 con 0x7f22a406dd20 2023-07-04T18:51:48.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.142+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2298018360 con 0x7f22a406dd20 2023-07-04T18:51:48.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.143+0000 7f2292ffd700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 0x7f228406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.143+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f22980890d0 con 0x7f22a406dd20 2023-07-04T18:51:48.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.143+0000 7f22a259c700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 0x7f228406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.143+0000 7f22a259c700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 0x7f228406d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f229400b340 tx=0x7f2294013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.159 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.147+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2298054170 con 0x7f22a406dd20 2023-07-04T18:51:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.337+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22a4159980 con 0x7f22a406dd20 2023-07-04T18:51:48.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.338+0000 7f2292ffd700 1 -- 172.21.15.77:0/749491032 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f22a4159980 con 0x7f22a406dd20 2023-07-04T18:51:48.340 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:48.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 msgr2=0x7f228406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 0x7f228406d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f229400b340 tx=0x7f2294013000 comp rx=0 tx=0).stop 2023-07-04T18:51:48.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 msgr2=0x7f22a415c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f229800c780 tx=0x7f229800cb40 comp rx=0 tx=0).stop 2023-07-04T18:51:48.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 shutdown_connections 2023-07-04T18:51:48.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f228406ace0 0x7f228406d190 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f22a4159280 0x7f22a41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22a406c770 0x7f22a414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 --2- 172.21.15.77:0/749491032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f22a406dd20 0x7f22a415c070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.345 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 >> 172.21.15.77:0/749491032 conn(0x7f22a406bba0 msgr2=0x7f22a40d00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 shutdown_connections 2023-07-04T18:51:48.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.340+0000 7f22a8cba700 1 -- 172.21.15.77:0/749491032 wait complete. 2023-07-04T18:51:48.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:48.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:48.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.477+0000 7f9fc28e2700 1 Processor -- start 2023-07-04T18:51:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.477+0000 7f9fc28e2700 1 -- start start 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.477+0000 7f9fc28e2700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9fbc1440b0 0x7f9fbc15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc28e2700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 0x7f9fbc0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc28e2700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc15f180 con 0x7f9fbc0ce8c0 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc28e2700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0cee30 con 0x7f9fbc1440b0 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc28e2700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0cf110 con 0x7f9fbc0d01d0 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9fbc1440b0 0x7f9fbc15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9fbc1440b0 0x7f9fbc15ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44834/0 (socket says 172.21.15.77:44834) 2023-07-04T18:51:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 -- 172.21.15.77:0/1660660952 learned_addr learned my addr 172.21.15.77:0/1660660952 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 -- 172.21.15.77:0/1660660952 >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 msgr2=0x7f9fbc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc0e7f700 1 --2- 172.21.15.77:0/1660660952 >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 0x7f9fbc0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/1660660952 >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 0x7f9fbc0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 -- 172.21.15.77:0/1660660952 >> v1:172.21.15.82:6789/0 conn(0x7f9fbc0ce8c0 legacy=0x7f9fbc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 -- 172.21.15.77:0/1660660952 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0cf3f0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fc0e7f700 1 --2- 172.21.15.77:0/1660660952 >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 0x7f9fbc0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/1660660952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15ec40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc071870 tx=0x7f9fac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7876ac618cb9ccd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbaffd700 1 -- 172.21.15.77:0/1660660952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac004230 con 0x7f9fbc1440b0 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbaffd700 1 -- 172.21.15.77:0/1660660952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac004390 con 0x7f9fbc1440b0 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.478+0000 7f9fbaffd700 1 -- 172.21.15.77:0/1660660952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0036a0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 -- 172.21.15.77:0/1660660952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 msgr2=0x7f9fbc15ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/1660660952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15ec40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc071870 tx=0x7f9fac009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 -- 172.21.15.77:0/1660660952 shutdown_connections 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/1660660952 >> v2:172.21.15.82:3300/0 conn(0x7f9fbc0d01d0 0x7f9fbc0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/1660660952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15ec40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 -- 172.21.15.77:0/1660660952 >> 172.21.15.77:0/1660660952 conn(0x7f9fbc06d900 msgr2=0x7f9fbc06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 -- 172.21.15.77:0/1660660952 shutdown_connections 2023-07-04T18:51:48.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.479+0000 7f9fc28e2700 1 -- 172.21.15.77:0/1660660952 wait complete. 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.480+0000 7f9fc28e2700 1 Processor -- start 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.480+0000 7f9fc28e2700 1 -- start start 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 0x7f9fbc1f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 0x7f9fbc158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1f2610 con 0x7f9fbc1440b0 2023-07-04T18:51:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1f28f0 con 0x7f9fbc0ce8c0 2023-07-04T18:51:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc28e2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1f2bd0 con 0x7f9fbc0d01d0 2023-07-04T18:51:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44842/0 (socket says 172.21.15.77:44842) 2023-07-04T18:51:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 -- 172.21.15.77:0/4015367985 learned_addr learned my addr 172.21.15.77:0/4015367985 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:48.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fbb7fe700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 0x7f9fbc158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 0x7f9fbc1f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 -- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 msgr2=0x7f9fbc158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 0x7f9fbc158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 -- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 msgr2=0x7f9fbc1f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 0x7f9fbc1f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 -- 172.21.15.77:0/4015367985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac009550 con 0x7f9fbc1440b0 2023-07-04T18:51:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fbb7fe700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 0x7f9fbc158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 0x7f9fbc1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.481+0000 7f9fc0e7f700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9fb400cea0 tx=0x7f9fb400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.482+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb400ae50 con 0x7f9fbc1440b0 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.482+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4003a20 con 0x7f9fbc1440b0 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.482+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc0c8fe0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.482+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4003dd0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.482+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc158540 con 0x7f9fbc1440b0 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.483+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc0c9220 con 0x7f9fbc1440b0 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.483+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9fb4003f30 con 0x7f9fbc1440b0 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.483+0000 7f9fb8ff9700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 0x7f9fa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.484+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9fb4089880 con 0x7f9fbc1440b0 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.484+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 0x7f9fa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.484+0000 7f9fbbfff700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 0x7f9fa806d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009130 tx=0x7f9fac008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.488+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9fb40549c0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.674+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9fbc04e210 con 0x7f9fbc1440b0 2023-07-04T18:51:48.676 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.674+0000 7f9fb8ff9700 1 -- 172.21.15.77:0/4015367985 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9fb4057fe0 con 0x7f9fbc1440b0 2023-07-04T18:51:48.676 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:48.678 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 msgr2=0x7f9fa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 0x7f9fa806d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009130 tx=0x7f9fac008030 comp rx=0 tx=0).stop 2023-07-04T18:51:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 msgr2=0x7f9fbc15aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.679 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9fb400cea0 tx=0x7f9fb400cf80 comp rx=0 tx=0).stop 2023-07-04T18:51:48.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 shutdown_connections 2023-07-04T18:51:48.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9fa806ace0 0x7f9fa806d190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9fbc1440b0 0x7f9fbc15aa10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fbc0ce8c0 0x7f9fbc1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 --2- 172.21.15.77:0/4015367985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9fbc0d01d0 0x7f9fbc158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 >> 172.21.15.77:0/4015367985 conn(0x7f9fbc06d900 msgr2=0x7f9fbc0c6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 shutdown_connections 2023-07-04T18:51:48.682 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.676+0000 7f9fc28e2700 1 -- 172.21.15.77:0/4015367985 wait complete. 2023-07-04T18:51:48.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:48.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.816+0000 7f68e7d5a700 1 Processor -- start 2023-07-04T18:51:48.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.816+0000 7f68e7d5a700 1 -- start start 2023-07-04T18:51:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e7d5a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68e0069e40 0x7f68e006a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e7d5a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f68e00aeb40 0x7f68e015db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e7d5a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e00cf750 con 0x7f68e00d01c0 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e7d5a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e00af060 con 0x7f68e0069e40 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e7d5a700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e00cf9c0 con 0x7f68e00aeb40 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e5af6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68e0069e40 0x7f68e006a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e52f5700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f68e00d01c0 0x7f68e00ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:51986/0 (socket says 172.21.15.77:51986) 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.817+0000 7f68e52f5700 1 -- 172.21.15.77:0/2110402275 learned_addr learned my addr 172.21.15.77:0/2110402275 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e62f7700 1 --2- 172.21.15.77:0/2110402275 >> v2:172.21.15.82:3300/0 conn(0x7f68e00aeb40 0x7f68e015db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e5af6700 1 -- 172.21.15.77:0/2110402275 >> v2:172.21.15.82:3300/0 conn(0x7f68e00aeb40 msgr2=0x7f68e015db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e5af6700 1 --2- 172.21.15.77:0/2110402275 >> v2:172.21.15.82:3300/0 conn(0x7f68e00aeb40 0x7f68e015db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e5af6700 1 -- 172.21.15.77:0/2110402275 >> v1:172.21.15.77:6789/0 conn(0x7f68e00d01c0 legacy=0x7f68e00ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e4af4700 1 -- 172.21.15.77:0/2110402275 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720749839 0 0) 0x7f68e00cf750 con 0x7f68e00d01c0 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e5af6700 1 -- 172.21.15.77:0/2110402275 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00cfca0 con 0x7f68e0069e40 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e5af6700 1 --2- 172.21.15.77:0/2110402275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e006a210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68e0071940 tx=0x7f68d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e531eba4cf5028 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e4af4700 1 -- 172.21.15.77:0/2110402275 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0011830 con 0x7f68e0069e40 2023-07-04T18:51:48.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e4af4700 1 -- 172.21.15.77:0/2110402275 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d0011990 con 0x7f68e0069e40 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.818+0000 7f68e4af4700 1 -- 172.21.15.77:0/2110402275 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0011d20 con 0x7f68e0069e40 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 -- 172.21.15.77:0/2110402275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 msgr2=0x7f68e006a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/2110402275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e006a210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f68e0071940 tx=0x7f68d0009b20 comp rx=0 tx=0).stop 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 -- 172.21.15.77:0/2110402275 shutdown_connections 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/2110402275 >> v2:172.21.15.82:3300/0 conn(0x7f68e00aeb40 0x7f68e015db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/2110402275 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e006a210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 -- 172.21.15.77:0/2110402275 >> 172.21.15.77:0/2110402275 conn(0x7f68e006af50 msgr2=0x7f68e006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 -- 172.21.15.77:0/2110402275 shutdown_connections 2023-07-04T18:51:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.819+0000 7f68e7d5a700 1 -- 172.21.15.77:0/2110402275 wait complete. 2023-07-04T18:51:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.820+0000 7f68e7d5a700 1 Processor -- start 2023-07-04T18:51:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.820+0000 7f68e7d5a700 1 -- start start 2023-07-04T18:51:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 0x7f68e00b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e00d01c0 0x7f68e01e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0141fa0 con 0x7f68e0069e40 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e01523e0 con 0x7f68e00d01c0 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e0153fa0 con 0x7f68e00aeb40 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44854/0 (socket says 172.21.15.77:44854) 2023-07-04T18:51:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 -- 172.21.15.77:0/3116633301 learned_addr learned my addr 172.21.15.77:0/3116633301 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e62f7700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e00d01c0 0x7f68e01e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e52f5700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 0x7f68e00b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 -- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 msgr2=0x7f68e00b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 0x7f68e00b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 -- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e00d01c0 msgr2=0x7f68e01e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e00d01c0 0x7f68e01e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:48.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 -- 172.21.15.77:0/3116633301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00094a0 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e52f5700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 0x7f68e00b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e5af6700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68d0014040 tx=0x7f68d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d00038a0 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d001e070 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e01e71f0 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0003c40 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.821+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e01e77b0 con 0x7f68e0069e40 2023-07-04T18:51:48.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.822+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e01e7050 con 0x7f68e0069e40 2023-07-04T18:51:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.823+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f68d002d450 con 0x7f68e0069e40 2023-07-04T18:51:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.823+0000 7f68d67fc700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.823+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f68d008a510 con 0x7f68e0069e40 2023-07-04T18:51:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.823+0000 7f68e52f5700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.824+0000 7f68e52f5700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 0x7f68c406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f60 tx=0x7f68dc007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:48.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:48.828+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f68e01e7050 con 0x7f68e0069e40 2023-07-04T18:51:49.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.011+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f68e01e7050 con 0x7f68e0069e40 2023-07-04T18:51:49.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.011+0000 7f68d67fc700 1 -- 172.21.15.77:0/3116633301 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f68d0020020 con 0x7f68e0069e40 2023-07-04T18:51:49.014 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 msgr2=0x7f68c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 0x7f68c406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f60 tx=0x7f68dc007ca0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 msgr2=0x7f68e01564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68d0014040 tx=0x7f68d0004750 comp rx=0 tx=0).stop 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 shutdown_connections 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f68c406ace0 0x7f68c406d190 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68e0069e40 0x7f68e01564d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f68e00d01c0 0x7f68e01e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 --2- 172.21.15.77:0/3116633301 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f68e00aeb40 0x7f68e00b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 >> 172.21.15.77:0/3116633301 conn(0x7f68e006af50 msgr2=0x7f68e006dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 shutdown_connections 2023-07-04T18:51:49.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.014+0000 7f68e7d5a700 1 -- 172.21.15.77:0/3116633301 wait complete. 2023-07-04T18:51:49.028 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:49.150 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.149+0000 7f9476c2b700 1 Processor -- start 2023-07-04T18:51:49.151 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.149+0000 7f9476c2b700 1 -- start start 2023-07-04T18:51:49.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f9476c2b700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f94701590b0 0x7f947015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f9476c2b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f94700b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.153 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f9476c2b700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94700b3f60 con 0x7f947006cfc0 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f9476c2b700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947006caa0 con 0x7f94701590b0 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f9476c2b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947015dc10 con 0x7f94700b3330 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f94751c8700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f947006cfc0 0x7f9470159480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52004/0 (socket says 172.21.15.77:52004) 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f94751c8700 1 -- 172.21.15.77:0/2944695391 learned_addr learned my addr 172.21.15.77:0/2944695391 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f946ffff700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f94700b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312945509 0 0) 0x7f94700b3f60 con 0x7f947006cfc0 2023-07-04T18:51:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f94749c7700 1 --2- 172.21.15.77:0/2944695391 >> v2:172.21.15.82:3300/0 conn(0x7f94701590b0 0x7f947015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.150+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94600035b0 con 0x7f947006cfc0 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296612143 0 0) 0x7f94600035b0 con 0x7f947006cfc0 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 msgr2=0x7f94700b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f94700b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 >> v2:172.21.15.82:3300/0 conn(0x7f94701590b0 msgr2=0x7f947015d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 --2- 172.21.15.77:0/2944695391 >> v2:172.21.15.82:3300/0 conn(0x7f94701590b0 0x7f947015d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947006c4e0 con 0x7f947006cfc0 2023-07-04T18:51:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f94680019f0 con 0x7f947006cfc0 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 >> v1:172.21.15.77:6789/0 conn(0x7f947006cfc0 legacy=0x7f9470159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9460003a00 0x7f9460003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f9460004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9460004d90 0x7f9460005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947006caa0 con 0x7f9460003a00 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947015dc10 con 0x7f9460004d90 2023-07-04T18:51:49.156 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946000ba10 con 0x7f94700b3330 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9460003a00 0x7f9460003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946ffff700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f9460004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94751c8700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9460004d90 0x7f9460005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 -- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 msgr2=0x7f9460004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94700b3330 0x7f9460004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 -- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9460004d90 msgr2=0x7f9460005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9460004d90 0x7f9460005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.157 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 -- 172.21.15.77:0/2944695391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600035b0 con 0x7f9460003a00 2023-07-04T18:51:49.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f94749c7700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9460003a00 0x7f9460003e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9464009b60 tx=0x7f9460005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65335e3c8a13e30a server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400e6c0 con 0x7f9460003a00 2023-07-04T18:51:49.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946400e820 con 0x7f9460003a00 2023-07-04T18:51:49.158 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.151+0000 7f946f7fe700 1 -- 172.21.15.77:0/2944695391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400eb90 con 0x7f9460003a00 2023-07-04T18:51:49.161 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 -- 172.21.15.77:0/2944695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9460003a00 msgr2=0x7f9460003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 --2- 172.21.15.77:0/2944695391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9460003a00 0x7f9460003e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9464009b60 tx=0x7f9460005880 comp rx=0 tx=0).stop 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f94751c8700 1 -- 172.21.15.77:0/2944695391 reap_dead start 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 -- 172.21.15.77:0/2944695391 shutdown_connections 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 -- 172.21.15.77:0/2944695391 >> 172.21.15.77:0/2944695391 conn(0x7f947006af50 msgr2=0x7f947006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 -- 172.21.15.77:0/2944695391 shutdown_connections 2023-07-04T18:51:49.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.152+0000 7f9476c2b700 1 -- 172.21.15.77:0/2944695391 wait complete. 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.154+0000 7f9476c2b700 1 Processor -- start 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.154+0000 7f9476c2b700 1 -- start start 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9460003a00 0x7f9470149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f947014fbc0 0x7f94700b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947014e500 con 0x7f947014f750 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947014d8f0 con 0x7f947014fbc0 2023-07-04T18:51:49.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f9476c2b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947014dbd0 con 0x7f9460003a00 2023-07-04T18:51:49.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44876/0 (socket says 172.21.15.77:44876) 2023-07-04T18:51:49.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 -- 172.21.15.77:0/334460063 learned_addr learned my addr 172.21.15.77:0/334460063 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:49.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f94749c7700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9460003a00 0x7f9470149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f94751c8700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f947014fbc0 0x7f94700b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 -- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9460003a00 msgr2=0x7f9470149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9460003a00 0x7f9470149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 -- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f947014fbc0 msgr2=0x7f94700b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f947014fbc0 0x7f94700b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.165 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 -- 172.21.15.77:0/334460063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947014deb0 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.155+0000 7f946ffff700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f945c011aa0 tx=0x7f945c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.156+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00f980 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.156+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c00fae0 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.156+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947014e190 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.156+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00fe90 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.156+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9470149ca0 con 0x7f947014f750 2023-07-04T18:51:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.157+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947014f440 con 0x7f947014f750 2023-07-04T18:51:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.157+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f945c014440 con 0x7f947014f750 2023-07-04T18:51:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.157+0000 7f946d7fa700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 0x7f945406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.157+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f945c089d20 con 0x7f947014f750 2023-07-04T18:51:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.158+0000 7f94749c7700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 0x7f945406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.158+0000 7f94749c7700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 0x7f945406d180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f94640090b0 tx=0x7f9464008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.162+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f945c058890 con 0x7f947014f750 2023-07-04T18:51:49.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.349+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f947004e210 con 0x7f947014f750 2023-07-04T18:51:49.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.349+0000 7f946d7fa700 1 -- 172.21.15.77:0/334460063 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f945c058420 con 0x7f947014f750 2023-07-04T18:51:49.351 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 msgr2=0x7f945406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 0x7f945406d180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f94640090b0 tx=0x7f9464008030 comp rx=0 tx=0).stop 2023-07-04T18:51:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 msgr2=0x7f94700b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f945c011aa0 tx=0x7f945c011e60 comp rx=0 tx=0).stop 2023-07-04T18:51:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 shutdown_connections 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f945406acd0 0x7f945406d180 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f947014fbc0 0x7f94700b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9460003a00 0x7f9470149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 --2- 172.21.15.77:0/334460063 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f947014f750 0x7f94700b13e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 >> 172.21.15.77:0/334460063 conn(0x7f947006af50 msgr2=0x7f94700af490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 shutdown_connections 2023-07-04T18:51:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.351+0000 7f9476c2b700 1 -- 172.21.15.77:0/334460063 wait complete. 2023-07-04T18:51:49.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:49.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.487+0000 7fcacdf9a700 1 Processor -- start 2023-07-04T18:51:49.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.487+0000 7fcacdf9a700 1 -- start start 2023-07-04T18:51:49.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcacdf9a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcac81590b0 0x7fcac815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcacdf9a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac806d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcacdf9a700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac80b3f60 con 0x7fcac80b3330 2023-07-04T18:51:49.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcacdf9a700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac806caa0 con 0x7fcac81590b0 2023-07-04T18:51:49.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcacdf9a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80b4b00 con 0x7fcac806cfc0 2023-07-04T18:51:49.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcac77fe700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcac81590b0 0x7fcac815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcac7fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcac6ffd700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fcac80b3330 0x7fcac80b43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:59932/0 (socket says 172.21.15.77:59932) 2023-07-04T18:51:49.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.488+0000 7fcac6ffd700 1 -- 172.21.15.77:0/2501552311 learned_addr learned my addr 172.21.15.77:0/2501552311 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:49.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac7fff700 1 -- 172.21.15.77:0/2501552311 >> v2:172.21.15.82:3300/0 conn(0x7fcac81590b0 msgr2=0x7fcac815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:49.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac7fff700 1 --2- 172.21.15.77:0/2501552311 >> v2:172.21.15.82:3300/0 conn(0x7fcac81590b0 0x7fcac815d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac7fff700 1 -- 172.21.15.77:0/2501552311 >> v1:172.21.15.82:6789/0 conn(0x7fcac80b3330 legacy=0x7fcac80b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac7fff700 1 -- 172.21.15.77:0/2501552311 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac806d940 con 0x7fcac806cfc0 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac7fff700 1 --2- 172.21.15.77:0/2501552311 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac806d400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3afc2285928d0ae4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac67fc700 1 -- 172.21.15.77:0/2501552311 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac000e070 con 0x7fcac806cfc0 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac67fc700 1 -- 172.21.15.77:0/2501552311 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac000bb20 con 0x7fcac806cfc0 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcac67fc700 1 -- 172.21.15.77:0/2501552311 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0012430 con 0x7fcac806cfc0 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 -- 172.21.15.77:0/2501552311 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 msgr2=0x7fcac806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/2501552311 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac806d400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac000b330 comp rx=0 tx=0).stop 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 -- 172.21.15.77:0/2501552311 shutdown_connections 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/2501552311 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac806d400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/2501552311 >> v2:172.21.15.82:3300/0 conn(0x7fcac81590b0 0x7fcac815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 -- 172.21.15.77:0/2501552311 >> 172.21.15.77:0/2501552311 conn(0x7fcac806af50 msgr2=0x7fcac806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.489+0000 7fcacdf9a700 1 -- 172.21.15.77:0/2501552311 shutdown_connections 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.490+0000 7fcacdf9a700 1 -- 172.21.15.77:0/2501552311 wait complete. 2023-07-04T18:51:49.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.491+0000 7fcacdf9a700 1 Processor -- start 2023-07-04T18:51:49.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.491+0000 7fcacdf9a700 1 -- start start 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.491+0000 7fcacdf9a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.491+0000 7fcacdf9a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcac80b3330 0x7fcac80a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.491+0000 7fcacdf9a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcacdf9a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80a98f0 con 0x7fcac81590b0 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcacdf9a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80a9bd0 con 0x7fcac80b3330 2023-07-04T18:51:49.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcacdf9a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80a9eb0 con 0x7fcac806cfc0 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44892/0 (socket says 172.21.15.77:44892) 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 -- 172.21.15.77:0/3546886444 learned_addr learned my addr 172.21.15.77:0/3546886444 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac77fe700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac6ffd700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcac80b3330 0x7fcac80a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 -- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 msgr2=0x7fcac815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 -- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcac80b3330 msgr2=0x7fcac80a8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcac80b3330 0x7fcac80a8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 -- 172.21.15.77:0/3546886444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac0009a70 con 0x7fcac81590b0 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcac7fff700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcac0014040 tx=0x7fcac0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0012430 con 0x7fcac81590b0 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac0020070 con 0x7fcac81590b0 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac80aa190 con 0x7fcac81590b0 2023-07-04T18:51:49.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0003ea0 con 0x7fcac81590b0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.492+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac8157900 con 0x7fcac81590b0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.493+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac804e210 con 0x7fcac81590b0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.494+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcac0004040 con 0x7fcac81590b0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.494+0000 7fcabffff700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 0x7fcaa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.494+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcac0022070 con 0x7fcac81590b0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.494+0000 7fcac77fe700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 0x7fcaa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.495+0000 7fcac77fe700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 0x7fcaa806d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009be0 tx=0x7fcab0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.499+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcac00561d0 con 0x7fcac81590b0 2023-07-04T18:51:49.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.684+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcac806e370 con 0x7fcac81590b0 2023-07-04T18:51:49.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.684+0000 7fcabffff700 1 -- 172.21.15.77:0/3546886444 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcac002c020 con 0x7fcac81590b0 2023-07-04T18:51:49.686 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:49.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 msgr2=0x7fcaa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 0x7fcaa806d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009be0 tx=0x7fcab0009210 comp rx=0 tx=0).stop 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 msgr2=0x7fcac8143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcac0014040 tx=0x7fcac0004ad0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 shutdown_connections 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcaa806ace0 0x7fcaa806d190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcac81590b0 0x7fcac8143040 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcac806cfc0 0x7fcac815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 --2- 172.21.15.77:0/3546886444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcac80b3330 0x7fcac80a8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 >> 172.21.15.77:0/3546886444 conn(0x7fcac806af50 msgr2=0x7fcac80af440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.686+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 shutdown_connections 2023-07-04T18:51:49.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.687+0000 7fcacdf9a700 1 -- 172.21.15.77:0/3546886444 wait complete. 2023-07-04T18:51:49.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:49.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.818+0000 7efe7c5ac700 1 Processor -- start 2023-07-04T18:51:49.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.818+0000 7efe7c5ac700 1 -- start start 2023-07-04T18:51:49.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7c5ac700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 0x7efe74069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7c5ac700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe740cb6d0 con 0x7efe74071150 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7c5ac700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe740af2e0 con 0x7efe740716a0 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7c5ac700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7415ee10 con 0x7efe740aef10 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7ab49700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efe74071150 0x7efe7415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52038/0 (socket says 172.21.15.77:52038) 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7ab49700 1 -- 172.21.15.77:0/1009732100 learned_addr learned my addr 172.21.15.77:0/1009732100 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe7a348700 1 --2- 172.21.15.77:0/1009732100 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 0x7efe74069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.822 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337195866 0 0) 0x7efe740cb6d0 con 0x7efe74071150 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.819+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe600035b0 con 0x7efe74071150 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe7a348700 1 -- 172.21.15.77:0/1009732100 >> v1:172.21.15.82:6789/0 conn(0x7efe740716a0 legacy=0x7efe7415dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe7a348700 1 -- 172.21.15.77:0/1009732100 >> v1:172.21.15.77:6789/0 conn(0x7efe74071150 legacy=0x7efe7415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048688073 0 0) 0x7efe740af2e0 con 0x7efe740716a0 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe7a348700 1 -- 172.21.15.77:0/1009732100 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74070d80 con 0x7efe740aef10 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe7a348700 1 --2- 172.21.15.77:0/1009732100 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 0x7efe74069e40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe64009870 tx=0x7efe64009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6f46afcf5692cfd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64011830 con 0x7efe740aef10 2023-07-04T18:51:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe64011990 con 0x7efe740aef10 2023-07-04T18:51:49.824 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.820+0000 7efe79346700 1 -- 172.21.15.77:0/1009732100 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64011d20 con 0x7efe740aef10 2023-07-04T18:51:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 -- 172.21.15.77:0/1009732100 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 msgr2=0x7efe74069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/1009732100 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 0x7efe74069e40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efe64009870 tx=0x7efe64009b80 comp rx=0 tx=0).stop 2023-07-04T18:51:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 -- 172.21.15.77:0/1009732100 shutdown_connections 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/1009732100 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740aef10 0x7efe74069e40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 -- 172.21.15.77:0/1009732100 >> 172.21.15.77:0/1009732100 conn(0x7efe7406bba0 msgr2=0x7efe740b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 -- 172.21.15.77:0/1009732100 shutdown_connections 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.821+0000 7efe7c5ac700 1 -- 172.21.15.77:0/1009732100 wait complete. 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.822+0000 7efe7c5ac700 1 Processor -- start 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.822+0000 7efe7c5ac700 1 -- start start 2023-07-04T18:51:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.822+0000 7efe7c5ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740716a0 0x7efe7415a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe740aef10 0x7efe7415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7415ce70 con 0x7efe74071150 2023-07-04T18:51:49.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740b29f0 con 0x7efe740aef10 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7415d680 con 0x7efe740716a0 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44902/0 (socket says 172.21.15.77:44902) 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 -- 172.21.15.77:0/4140989123 learned_addr learned my addr 172.21.15.77:0/4140989123 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 -- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740716a0 msgr2=0x7efe7415a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740716a0 0x7efe7415a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 -- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe740aef10 msgr2=0x7efe7415acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe740aef10 0x7efe7415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 -- 172.21.15.77:0/4140989123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe640094a0 con 0x7efe74071150 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7a348700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efe740d0ac0 tx=0x7efe64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6401bb30 con 0x7efe74071150 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6401bc90 con 0x7efe74071150 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640253f0 con 0x7efe74071150 2023-07-04T18:51:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7415d0c0 con 0x7efe74071150 2023-07-04T18:51:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.823+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7415bcf0 con 0x7efe74071150 2023-07-04T18:51:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.824+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7415bee0 con 0x7efe74071150 2023-07-04T18:51:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.825+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efe64025570 con 0x7efe74071150 2023-07-04T18:51:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.825+0000 7efe6affd700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 0x7efe5806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.825+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efe6408a9b0 con 0x7efe74071150 2023-07-04T18:51:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.825+0000 7efe79b47700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 0x7efe5806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.826+0000 7efe79b47700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 0x7efe5806d190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efe70007fd0 tx=0x7efe700079a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:49.829+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efe64055c10 con 0x7efe74071150 2023-07-04T18:51:50.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.017+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe7404e210 con 0x7efe74071150 2023-07-04T18:51:50.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.018+0000 7efe6affd700 1 -- 172.21.15.77:0/4140989123 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7efe64015090 con 0x7efe74071150 2023-07-04T18:51:50.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:50.020 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:50.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 msgr2=0x7efe5806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 0x7efe5806d190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efe70007fd0 tx=0x7efe700079a0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 msgr2=0x7efe74151800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efe740d0ac0 tx=0x7efe64004750 comp rx=0 tx=0).stop 2023-07-04T18:51:50.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 shutdown_connections 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efe5806ace0 0x7efe5806d190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe74071150 0x7efe74151800 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe740aef10 0x7efe7415acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 --2- 172.21.15.77:0/4140989123 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efe740716a0 0x7efe7415a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 >> 172.21.15.77:0/4140989123 conn(0x7efe7406bba0 msgr2=0x7efe74159ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 shutdown_connections 2023-07-04T18:51:50.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.020+0000 7efe7c5ac700 1 -- 172.21.15.77:0/4140989123 wait complete. 2023-07-04T18:51:50.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:50.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.159+0000 7f4900d29700 1 Processor -- start 2023-07-04T18:51:50.161 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.159+0000 7f4900d29700 1 -- start start 2023-07-04T18:51:50.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.160+0000 7f4900d29700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.160+0000 7f4900d29700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48fc0b3330 0x7f48fc0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.160+0000 7f4900d29700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0b3f60 con 0x7f48fc06cfc0 2023-07-04T18:51:50.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.160+0000 7f4900d29700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48fc06caa0 con 0x7f48fc0b3330 2023-07-04T18:51:50.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.160+0000 7f4900d29700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc15dc10 con 0x7f48fc1590b0 2023-07-04T18:51:50.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48fc0b3330 0x7f48fc0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48fad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48fc06cfc0 0x7f48fc159480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52050/0 (socket says 172.21.15.77:52050) 2023-07-04T18:51:50.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48fc0b3330 0x7f48fc0b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44910/0 (socket says 172.21.15.77:44910) 2023-07-04T18:51:50.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48fad9d700 1 -- 172.21.15.77:0/2307804416 learned_addr learned my addr 172.21.15.77:0/2307804416 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:50.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2307804416 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 msgr2=0x7f48fc15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2307804416 >> v1:172.21.15.77:6789/0 conn(0x7f48fc06cfc0 legacy=0x7f48fc159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2307804416 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc06c4e0 con 0x7f48fc0b3330 2023-07-04T18:51:50.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f959a700 1 -- 172.21.15.77:0/2307804416 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558144838 0 0) 0x7f48fc0b3f60 con 0x7f48fc06cfc0 2023-07-04T18:51:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:50.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f9d9b700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc0b43f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009840 tx=0x7f48ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123b50fd9f977621 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f959a700 1 -- 172.21.15.77:0/2307804416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011830 con 0x7f48fc0b3330 2023-07-04T18:51:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f959a700 1 -- 172.21.15.77:0/2307804416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec011990 con 0x7f48fc0b3330 2023-07-04T18:51:50.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.161+0000 7f48f959a700 1 -- 172.21.15.77:0/2307804416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011d20 con 0x7f48fc0b3330 2023-07-04T18:51:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 -- 172.21.15.77:0/2307804416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 msgr2=0x7f48fc0b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc0b43f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009840 tx=0x7f48ec009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 -- 172.21.15.77:0/2307804416 shutdown_connections 2023-07-04T18:51:50.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 --2- 172.21.15.77:0/2307804416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc0b43f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 -- 172.21.15.77:0/2307804416 >> 172.21.15.77:0/2307804416 conn(0x7f48fc06af50 msgr2=0x7f48fc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 -- 172.21.15.77:0/2307804416 shutdown_connections 2023-07-04T18:51:50.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.162+0000 7f4900d29700 1 -- 172.21.15.77:0/2307804416 wait complete. 2023-07-04T18:51:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.163+0000 7f4900d29700 1 Processor -- start 2023-07-04T18:51:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.163+0000 7f4900d29700 1 -- start start 2023-07-04T18:51:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 0x7f48fc145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc1533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc1f2a60 con 0x7f48fc0b3330 2023-07-04T18:51:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0aca50 con 0x7f48fc06cfc0 2023-07-04T18:51:50.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f4900d29700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0acd30 con 0x7f48fc1590b0 2023-07-04T18:51:50.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44916/0 (socket says 172.21.15.77:44916) 2023-07-04T18:51:50.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2876521985 learned_addr learned my addr 172.21.15.77:0/2876521985 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:50.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48fad9d700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc1533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 0x7f48fc145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 msgr2=0x7f48fc1533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc1533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 msgr2=0x7f48fc145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 0x7f48fc145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48f9d9b700 1 -- 172.21.15.77:0/2876521985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec0094a0 con 0x7f48fc0b3330 2023-07-04T18:51:50.182 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.164+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 0x7f48fc145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:50.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f48f9d9b700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0090b0 tx=0x7f48ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec0036a0 con 0x7f48fc0b3330 2023-07-04T18:51:50.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec01f070 con 0x7f48fc0b3330 2023-07-04T18:51:50.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc0ad010 con 0x7f48fc0b3330 2023-07-04T18:51:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec003a00 con 0x7f48fc0b3330 2023-07-04T18:51:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.165+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc1538f0 con 0x7f48fc0b3330 2023-07-04T18:51:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.166+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc0ad220 con 0x7f48fc0b3330 2023-07-04T18:51:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.166+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f48ec003b60 con 0x7f48fc0b3330 2023-07-04T18:51:50.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.167+0000 7f48eaffd700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 0x7f48dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.167+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f48ec021070 con 0x7f48fc0b3330 2023-07-04T18:51:50.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.167+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 0x7f48dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.167+0000 7f48fa59c700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 0x7f48dc06d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f48f000c960 tx=0x7f48f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.171+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f48ec055f40 con 0x7f48fc0b3330 2023-07-04T18:51:50.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.358+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48fc04e210 con 0x7f48fc0b3330 2023-07-04T18:51:50.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.358+0000 7f48eaffd700 1 -- 172.21.15.77:0/2876521985 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48ec0150a0 con 0x7f48fc0b3330 2023-07-04T18:51:50.360 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:50.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 msgr2=0x7f48dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 0x7f48dc06d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f48f000c960 tx=0x7f48f0008040 comp rx=0 tx=0).stop 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 msgr2=0x7f48fc144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0090b0 tx=0x7f48ec004750 comp rx=0 tx=0).stop 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 shutdown_connections 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f48dc06ace0 0x7f48dc06d190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f48fc1590b0 0x7f48fc1533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48fc06cfc0 0x7f48fc145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 --2- 172.21.15.77:0/2876521985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48fc0b3330 0x7f48fc144a40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.360+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 >> 172.21.15.77:0/2876521985 conn(0x7f48fc06af50 msgr2=0x7f48fc06b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.361+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 shutdown_connections 2023-07-04T18:51:50.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.361+0000 7f4900d29700 1 -- 172.21.15.77:0/2876521985 wait complete. 2023-07-04T18:51:50.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:50.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.496+0000 7efdee597700 1 Processor -- start 2023-07-04T18:51:50.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.496+0000 7efdee597700 1 -- start start 2023-07-04T18:51:50.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efdee597700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efde815a0c0 0x7efde806e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efdee597700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde806c3b0 0x7efde806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efdee597700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde80ce770 con 0x7efde815a490 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efdee597700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde80cf170 con 0x7efde815a0c0 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efdee597700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde80cf450 con 0x7efde806c3b0 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efde7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efde815a0c0 0x7efde806e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efde7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efde815a0c0 0x7efde806e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44922/0 (socket says 172.21.15.77:44922) 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.497+0000 7efde7fff700 1 -- 172.21.15.77:0/813215606 learned_addr learned my addr 172.21.15.77:0/813215606 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efdecb34700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde806c3b0 0x7efde806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde7fff700 1 -- 172.21.15.77:0/813215606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde806c3b0 msgr2=0x7efde806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde7fff700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde806c3b0 0x7efde806c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde7fff700 1 -- 172.21.15.77:0/813215606 >> v1:172.21.15.82:6789/0 conn(0x7efde815a490 legacy=0x7efde80cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde7fff700 1 -- 172.21.15.77:0/813215606 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80cf730 con 0x7efde815a0c0 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde7fff700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde815a0c0 0x7efde806e900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efdd80098f0 tx=0x7efdd8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cbdce696f87418 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde6ffd700 1 -- 172.21.15.77:0/813215606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800b8a0 con 0x7efde815a0c0 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde6ffd700 1 -- 172.21.15.77:0/813215606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd800ba00 con 0x7efde815a0c0 2023-07-04T18:51:50.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.498+0000 7efde6ffd700 1 -- 172.21.15.77:0/813215606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800bd90 con 0x7efde815a0c0 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 -- 172.21.15.77:0/813215606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde815a0c0 msgr2=0x7efde806e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde815a0c0 0x7efde806e900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efdd80098f0 tx=0x7efdd8009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 -- 172.21.15.77:0/813215606 shutdown_connections 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde806c3b0 0x7efde806c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 --2- 172.21.15.77:0/813215606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde815a0c0 0x7efde806e900 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 -- 172.21.15.77:0/813215606 >> 172.21.15.77:0/813215606 conn(0x7efde806af50 msgr2=0x7efde806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 -- 172.21.15.77:0/813215606 shutdown_connections 2023-07-04T18:51:50.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.499+0000 7efdee597700 1 -- 172.21.15.77:0/813215606 wait complete. 2023-07-04T18:51:50.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.500+0000 7efdee597700 1 Processor -- start 2023-07-04T18:51:50.506 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.500+0000 7efdee597700 1 -- start start 2023-07-04T18:51:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efde815a0c0 0x7efde8143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde815a490 0x7efde80c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde80ca410 con 0x7efde806c3b0 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde80cade0 con 0x7efde815a0c0 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde80c9800 con 0x7efde815a490 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44928/0 (socket says 172.21.15.77:44928) 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 -- 172.21.15.77:0/922571818 learned_addr learned my addr 172.21.15.77:0/922571818 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:50.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdecb34700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde815a490 0x7efde80c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde77fe700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efde815a0c0 0x7efde8143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 -- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde815a490 msgr2=0x7efde80c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde815a490 0x7efde80c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 -- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efde815a0c0 msgr2=0x7efde8143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efde815a0c0 0x7efde8143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 -- 172.21.15.77:0/922571818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009550 con 0x7efde806c3b0 2023-07-04T18:51:50.509 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde7fff700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efdd8000c00 tx=0x7efdd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8004480 con 0x7efde806c3b0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd801f070 con 0x7efde806c3b0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde80c9ae0 con 0x7efde806c3b0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd80038a0 con 0x7efde806c3b0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.501+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde80ca0a0 con 0x7efde806c3b0 2023-07-04T18:51:50.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.502+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde80ca260 con 0x7efde806c3b0 2023-07-04T18:51:50.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.503+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efdd8021030 con 0x7efde806c3b0 2023-07-04T18:51:50.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.503+0000 7efde4ff9700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 0x7efdd406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.503+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efdd808a8d0 con 0x7efde806c3b0 2023-07-04T18:51:50.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.503+0000 7efde77fe700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 0x7efdd406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.504+0000 7efde77fe700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 0x7efdd406d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efde814b4b0 tx=0x7efddc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.508+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efdd8055980 con 0x7efde806c3b0 2023-07-04T18:51:50.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.696+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efde804e210 con 0x7efde806c3b0 2023-07-04T18:51:50.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.697+0000 7efde4ff9700 1 -- 172.21.15.77:0/922571818 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7efdd8058fa0 con 0x7efde806c3b0 2023-07-04T18:51:50.702 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 msgr2=0x7efdd406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 0x7efdd406d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efde814b4b0 tx=0x7efddc009040 comp rx=0 tx=0).stop 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 msgr2=0x7efde80ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efdd8000c00 tx=0x7efdd8004750 comp rx=0 tx=0).stop 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 shutdown_connections 2023-07-04T18:51:50.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdd406ace0 0x7efdd406d190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde806c3b0 0x7efde80ca8a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efde815a490 0x7efde80c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 --2- 172.21.15.77:0/922571818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efde815a0c0 0x7efde8143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 >> 172.21.15.77:0/922571818 conn(0x7efde806af50 msgr2=0x7efde806dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 shutdown_connections 2023-07-04T18:51:50.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.699+0000 7efdee597700 1 -- 172.21.15.77:0/922571818 wait complete. 2023-07-04T18:51:50.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:50.836 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.834+0000 7f39551b9700 1 Processor -- start 2023-07-04T18:51:50.837 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.834+0000 7f39551b9700 1 -- start start 2023-07-04T18:51:50.837 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f39551b9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3950069e40 0x7f395006a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.837 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f39551b9700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 0x7f39500ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f39551b9700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39500cf750 con 0x7f39500ae970 2023-07-04T18:51:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f39551b9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39500af060 con 0x7f3950069e40 2023-07-04T18:51:50.840 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f39551b9700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39500cf9c0 con 0x7f39500d01c0 2023-07-04T18:51:50.840 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f394ed9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3950069e40 0x7f395006a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.840 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f394ed9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3950069e40 0x7f395006a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44936/0 (socket says 172.21.15.77:44936) 2023-07-04T18:51:50.840 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.835+0000 7f394ed9d700 1 -- 172.21.15.77:0/2097988321 learned_addr learned my addr 172.21.15.77:0/2097988321 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:50.843 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394e59c700 1 --2- 172.21.15.77:0/2097988321 >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 0x7f39500ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.844 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394ed9d700 1 -- 172.21.15.77:0/2097988321 >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 msgr2=0x7f39500ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:50.844 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394ed9d700 1 --2- 172.21.15.77:0/2097988321 >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 0x7f39500ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.844 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394ed9d700 1 -- 172.21.15.77:0/2097988321 >> v1:172.21.15.82:6789/0 conn(0x7f39500ae970 legacy=0x7f395015db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.844 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394ed9d700 1 -- 172.21.15.77:0/2097988321 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39500cfca0 con 0x7f3950069e40 2023-07-04T18:51:50.844 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394e59c700 1 --2- 172.21.15.77:0/2097988321 >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 0x7f39500ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394ed9d700 1 --2- 172.21.15.77:0/2097988321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3950069e40 0x7f395006a210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3950070bf0 tx=0x7f3944009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530c1d6c349062a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394dd9b700 1 -- 172.21.15.77:0/2097988321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394400b870 con 0x7f3950069e40 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394dd9b700 1 -- 172.21.15.77:0/2097988321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394400b9d0 con 0x7f3950069e40 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f394dd9b700 1 -- 172.21.15.77:0/2097988321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394400bd60 con 0x7f3950069e40 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f39551b9700 1 -- 172.21.15.77:0/2097988321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3950069e40 msgr2=0x7f395006a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.836+0000 7f39551b9700 1 --2- 172.21.15.77:0/2097988321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3950069e40 0x7f395006a210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3950070bf0 tx=0x7f3944009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 -- 172.21.15.77:0/2097988321 shutdown_connections 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 --2- 172.21.15.77:0/2097988321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3950069e40 0x7f395006a210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 --2- 172.21.15.77:0/2097988321 >> v2:172.21.15.82:3300/0 conn(0x7f39500d01c0 0x7f39500ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 -- 172.21.15.77:0/2097988321 >> 172.21.15.77:0/2097988321 conn(0x7f395006af50 msgr2=0x7f395006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 -- 172.21.15.77:0/2097988321 shutdown_connections 2023-07-04T18:51:50.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.837+0000 7f39551b9700 1 -- 172.21.15.77:0/2097988321 wait complete. 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.838+0000 7f39551b9700 1 Processor -- start 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.838+0000 7f39551b9700 1 -- start start 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3950069e40 0x7f39501e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39500d01c0 0x7f3950155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39501e5e90 con 0x7f39500ae970 2023-07-04T18:51:50.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395015db90 con 0x7f3950069e40 2023-07-04T18:51:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39501f2400 con 0x7f39500d01c0 2023-07-04T18:51:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44942/0 (socket says 172.21.15.77:44942) 2023-07-04T18:51:50.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 -- 172.21.15.77:0/3590908952 learned_addr learned my addr 172.21.15.77:0/3590908952 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394ed9d700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3950069e40 0x7f39501e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394f59e700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39500d01c0 0x7f3950155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 -- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39500d01c0 msgr2=0x7f3950155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39500d01c0 0x7f3950155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 -- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3950069e40 msgr2=0x7f39501e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3950069e40 0x7f39501e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 -- 172.21.15.77:0/3590908952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944009550 con 0x7f39500ae970 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f394e59c700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f394000e810 tx=0x7f394000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394000c070 con 0x7f39500ae970 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39501f26e0 con 0x7f39500ae970 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3940004230 con 0x7f39500ae970 2023-07-04T18:51:50.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.839+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39400036a0 con 0x7f39500ae970 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.840+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39501f2ca0 con 0x7f39500ae970 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.840+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39500acb30 con 0x7f39500ae970 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.841+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3940003800 con 0x7f39500ae970 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.841+0000 7f39377fe700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 0x7f393c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.841+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f3940089170 con 0x7f39500ae970 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.842+0000 7f394ed9d700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 0x7f393c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:50.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.842+0000 7f394ed9d700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 0x7f393c06d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3944014040 tx=0x7f3944008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:50.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:50.846+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3940054230 con 0x7f39500ae970 2023-07-04T18:51:51.032 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.031+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f395004e210 con 0x7f39500ae970 2023-07-04T18:51:51.033 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.031+0000 7f39377fe700 1 -- 172.21.15.77:0/3590908952 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f39400094c0 con 0x7f39500ae970 2023-07-04T18:51:51.033 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 msgr2=0x7f393c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 0x7f393c06d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3944014040 tx=0x7f3944008030 comp rx=0 tx=0).stop 2023-07-04T18:51:51.035 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 msgr2=0x7f395015de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f394000e810 tx=0x7f394000eb20 comp rx=0 tx=0).stop 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 shutdown_connections 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f393c06ace0 0x7f393c06d190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39500d01c0 0x7f3950155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3950069e40 0x7f39501e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 --2- 172.21.15.77:0/3590908952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39500ae970 0x7f395015de70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 >> 172.21.15.77:0/3590908952 conn(0x7f395006af50 msgr2=0x7f3950160350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 shutdown_connections 2023-07-04T18:51:51.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.033+0000 7f39551b9700 1 -- 172.21.15.77:0/3590908952 wait complete. 2023-07-04T18:51:51.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:51.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.168+0000 7fe4fa31b700 1 Processor -- start 2023-07-04T18:51:51.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.168+0000 7fe4fa31b700 1 -- start start 2023-07-04T18:51:51.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.169+0000 7fe4fa31b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f41440b0 0x7fe4f415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.169+0000 7fe4fa31b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4f40d01d0 0x7fe4f40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.169+0000 7fe4fa31b700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f415f180 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.169+0000 7fe4fa31b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f40cee30 con 0x7fe4f40d01d0 2023-07-04T18:51:51.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.169+0000 7fe4fa31b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40cf110 con 0x7fe4f41440b0 2023-07-04T18:51:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4f40d01d0 0x7fe4f40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4f40d01d0 0x7fe4f40cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44952/0 (socket says 172.21.15.77:44952) 2023-07-04T18:51:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 -- 172.21.15.77:0/777870575 learned_addr learned my addr 172.21.15.77:0/777870575 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:51.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f3fff700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f41440b0 0x7fe4f415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 -- 172.21.15.77:0/777870575 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f41440b0 msgr2=0x7fe4f415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f41440b0 0x7fe4f415ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 -- 172.21.15.77:0/777870575 >> v1:172.21.15.82:6789/0 conn(0x7fe4f40ce8c0 legacy=0x7fe4f40cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:51.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 -- 172.21.15.77:0/777870575 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40cf3f0 con 0x7fe4f40d01d0 2023-07-04T18:51:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f88b8700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40d01d0 0x7fe4f40cc730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009e50 tx=0x7fe4ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f101ec40a677157e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f2ffd700 1 -- 172.21.15.77:0/777870575 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec00e070 con 0x7fe4f40d01d0 2023-07-04T18:51:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f2ffd700 1 -- 172.21.15.77:0/777870575 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec003a20 con 0x7fe4f40d01d0 2023-07-04T18:51:51.183 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.170+0000 7fe4f2ffd700 1 -- 172.21.15.77:0/777870575 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec003dd0 con 0x7fe4f40d01d0 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 -- 172.21.15.77:0/777870575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40d01d0 msgr2=0x7fe4f40cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40d01d0 0x7fe4f40cc730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec009e50 tx=0x7fe4ec00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 -- 172.21.15.77:0/777870575 shutdown_connections 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40d01d0 0x7fe4f40cc730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/777870575 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f41440b0 0x7fe4f415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 -- 172.21.15.77:0/777870575 >> 172.21.15.77:0/777870575 conn(0x7fe4f406d900 msgr2=0x7fe4f406dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 -- 172.21.15.77:0/777870575 shutdown_connections 2023-07-04T18:51:51.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.171+0000 7fe4fa31b700 1 -- 172.21.15.77:0/777870575 wait complete. 2023-07-04T18:51:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.172+0000 7fe4fa31b700 1 Processor -- start 2023-07-04T18:51:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.172+0000 7fe4fa31b700 1 -- start start 2023-07-04T18:51:51.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f40d01d0 0x7fe4f415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 0x7fe4f415bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40a9b70 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f4158670 con 0x7fe4f41440b0 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40a8f60 con 0x7fe4f40d01d0 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44954/0 (socket says 172.21.15.77:44954) 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 -- 172.21.15.77:0/1392563398 learned_addr learned my addr 172.21.15.77:0/1392563398 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f37fe700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f40d01d0 0x7fe4f415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f88b8700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 0x7fe4f415bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 -- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f40d01d0 msgr2=0x7fe4f415ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.189 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f40d01d0 0x7fe4f415ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 -- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 msgr2=0x7fe4f415bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 0x7fe4f415bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 -- 172.21.15.77:0/1392563398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec009b30 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f88b8700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 0x7fe4f415bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f3fff700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800c820 tx=0x7fe4e800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011800 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8011960 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f40a9240 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011cf0 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.173+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f40a9800 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.174+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f404e210 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.175+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe4e8018360 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.175+0000 7fe4f0ff9700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 0x7fe4e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.175+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe4e80893d0 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.191 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.175+0000 7fe4f37fe700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 0x7fe4e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.176+0000 7fe4f37fe700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 0x7fe4e006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4159ab0 tx=0x7fe4e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.192 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.179+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe4e8054490 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:51:51.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:51:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.375+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4f40ce450 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.375+0000 7fe4f0ff9700 1 -- 172.21.15.77:0/1392563398 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4e8057ab0 con 0x7fe4f40ce8c0 2023-07-04T18:51:51.377 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 msgr2=0x7fe4e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 0x7fe4e006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4159ab0 tx=0x7fe4e4009040 comp rx=0 tx=0).stop 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 msgr2=0x7fe4f40a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800c820 tx=0x7fe4e800cbe0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 shutdown_connections 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe4e006ace0 0x7fe4e006d190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4f40ce8c0 0x7fe4f40a84a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4f41440b0 0x7fe4f415bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 --2- 172.21.15.77:0/1392563398 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe4f40d01d0 0x7fe4f415ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.377+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 >> 172.21.15.77:0/1392563398 conn(0x7fe4f406d900 msgr2=0x7fe4f40c9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.378+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 shutdown_connections 2023-07-04T18:51:51.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.378+0000 7fe4fa31b700 1 -- 172.21.15.77:0/1392563398 wait complete. 2023-07-04T18:51:51.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:51.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.511+0000 7f4527484700 1 Processor -- start 2023-07-04T18:51:51.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.511+0000 7f4527484700 1 -- start start 2023-07-04T18:51:51.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4527484700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f452006dd20 0x7f45200cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4527484700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452006d890 con 0x7f45200cdad0 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4527484700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452006cc80 con 0x7f4520159280 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4527484700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45200cdf70 con 0x7f452006dd20 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4524a1f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f452006dd20 0x7f45200cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4525a21700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f45200cdad0 0x7f45200ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52104/0 (socket says 172.21.15.77:52104) 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4525a21700 1 -- 172.21.15.77:0/3877812135 learned_addr learned my addr 172.21.15.77:0/3877812135 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:51.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4524a1f700 1 -- 172.21.15.77:0/3877812135 >> v1:172.21.15.82:6789/0 conn(0x7f4520159280 legacy=0x7f452006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4524a1f700 1 -- 172.21.15.77:0/3877812135 >> v1:172.21.15.77:6789/0 conn(0x7f45200cdad0 legacy=0x7f45200ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4524a1f700 1 -- 172.21.15.77:0/3877812135 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452006cf60 con 0x7f452006dd20 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4524a1f700 1 --2- 172.21.15.77:0/3877812135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452006dd20 0x7f45200cd590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4510009840 tx=0x7f4510009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb90b329239bd81 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.512+0000 7f4517fff700 1 -- 172.21.15.77:0/3877812135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510004230 con 0x7f452006dd20 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4517fff700 1 -- 172.21.15.77:0/3877812135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4510004390 con 0x7f452006dd20 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4517fff700 1 -- 172.21.15.77:0/3877812135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45100036a0 con 0x7f452006dd20 2023-07-04T18:51:51.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 -- 172.21.15.77:0/3877812135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452006dd20 msgr2=0x7f45200cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 --2- 172.21.15.77:0/3877812135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452006dd20 0x7f45200cd590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4510009840 tx=0x7f4510009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:51.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 -- 172.21.15.77:0/3877812135 shutdown_connections 2023-07-04T18:51:51.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 --2- 172.21.15.77:0/3877812135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452006dd20 0x7f45200cd590 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 -- 172.21.15.77:0/3877812135 >> 172.21.15.77:0/3877812135 conn(0x7f452006bba0 msgr2=0x7f45200b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 -- 172.21.15.77:0/3877812135 shutdown_connections 2023-07-04T18:51:51.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.513+0000 7f4527484700 1 -- 172.21.15.77:0/3877812135 wait complete. 2023-07-04T18:51:51.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.514+0000 7f4527484700 1 Processor -- start 2023-07-04T18:51:51.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.514+0000 7f4527484700 1 -- start start 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452006dd20 0x7f452014c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 0x7f45201f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201f3190 con 0x7f4520159280 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452014ace0 con 0x7f452006dd20 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4527484700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452014cae0 con 0x7f45200cdad0 2023-07-04T18:51:51.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.523 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44972/0 (socket says 172.21.15.77:44972) 2023-07-04T18:51:51.523 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 -- 172.21.15.77:0/4098675434 learned_addr learned my addr 172.21.15.77:0/4098675434 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525220700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452006dd20 0x7f452014c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4524a1f700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 0x7f45201f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 -- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 msgr2=0x7f45201f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 0x7f45201f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 -- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452006dd20 msgr2=0x7f452014c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452006dd20 0x7f452014c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.524 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4525a21700 1 -- 172.21.15.77:0/4098675434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100094a0 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.515+0000 7f4524a1f700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 0x7f45201f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4525a21700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f451800ee60 tx=0x7f451800c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518011470 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45180115d0 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452014cdc0 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518011940 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.516+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452014d380 con 0x7f4520159280 2023-07-04T18:51:51.525 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.517+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452004e210 con 0x7f4520159280 2023-07-04T18:51:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.517+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4518011aa0 con 0x7f4520159280 2023-07-04T18:51:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.517+0000 7f4515ffb700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 0x7f450406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.526 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.518+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f451808c030 con 0x7f4520159280 2023-07-04T18:51:51.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.518+0000 7f4525220700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 0x7f450406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.518+0000 7f4525220700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 0x7f450406d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f451c009960 tx=0x7f451c0090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.532 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.522+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f45180570e0 con 0x7f4520159280 2023-07-04T18:51:51.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.708+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45200734d0 con 0x7f4520159280 2023-07-04T18:51:51.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.709+0000 7f4515ffb700 1 -- 172.21.15.77:0/4098675434 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f451805a700 con 0x7f4520159280 2023-07-04T18:51:51.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:51.712 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:51.713 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.711+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 msgr2=0x7f450406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.711+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 0x7f450406d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f451c009960 tx=0x7f451c0090e0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.711+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 msgr2=0x7f45200707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.711+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f451800ee60 tx=0x7f451800c550 comp rx=0 tx=0).stop 2023-07-04T18:51:51.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 shutdown_connections 2023-07-04T18:51:51.714 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f450406ace0 0x7f450406d190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4520159280 0x7f45200707d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452006dd20 0x7f452014c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 --2- 172.21.15.77:0/4098675434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f45200cdad0 0x7f45201f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 >> 172.21.15.77:0/4098675434 conn(0x7f452006bba0 msgr2=0x7f45200713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 shutdown_connections 2023-07-04T18:51:51.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.712+0000 7f4527484700 1 -- 172.21.15.77:0/4098675434 wait complete. 2023-07-04T18:51:51.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:51.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:51.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.859+0000 7f7ccec24700 1 Processor -- start 2023-07-04T18:51:51.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.859+0000 7f7ccec24700 1 -- start start 2023-07-04T18:51:51.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccec24700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 0x7f7cc80ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccec24700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc81606d0 con 0x7f7cc80b0360 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccec24700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc80cf210 con 0x7f7cc80cd700 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccec24700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80cf7d0 con 0x7f7cc80b2800 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccc9c0700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7cc80b0360 0x7f7cc8160b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52114/0 (socket says 172.21.15.77:52114) 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.860+0000 7f7ccc9c0700 1 -- 172.21.15.77:0/3816044834 learned_addr learned my addr 172.21.15.77:0/3816044834 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:51.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc7fff700 1 --2- 172.21.15.77:0/3816044834 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 0x7f7cc80ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589098980 0 0) 0x7f7cc81606d0 con 0x7f7cc80b0360 2023-07-04T18:51:51.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb80035b0 con 0x7f7cc80b0360 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631758987 0 0) 0x7f7cc80cf210 con 0x7f7cc80cd700 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc81606d0 con 0x7f7cc80cd700 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3816044834 >> v1:172.21.15.82:6789/0 conn(0x7f7cc80cd700 legacy=0x7f7cc80cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3816044834 >> v1:172.21.15.77:6789/0 conn(0x7f7cc80b0360 legacy=0x7f7cc8160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3816044834 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80cf4f0 con 0x7f7cc80b2800 2023-07-04T18:51:51.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc7fff700 1 --2- 172.21.15.77:0/3816044834 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 0x7f7cc80ce8c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009870 tx=0x7f7cb4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f4c8c47d805476 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4011830 con 0x7f7cc80b2800 2023-07-04T18:51:51.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4011990 con 0x7f7cc80b2800 2023-07-04T18:51:51.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.861+0000 7f7cc77fe700 1 -- 172.21.15.77:0/3816044834 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4011d00 con 0x7f7cc80b2800 2023-07-04T18:51:51.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 -- 172.21.15.77:0/3816044834 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 msgr2=0x7f7cc80ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3816044834 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 0x7f7cc80ce8c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009870 tx=0x7f7cb4009b80 comp rx=0 tx=0).stop 2023-07-04T18:51:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 -- 172.21.15.77:0/3816044834 shutdown_connections 2023-07-04T18:51:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3816044834 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80b2800 0x7f7cc80ce8c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 -- 172.21.15.77:0/3816044834 >> 172.21.15.77:0/3816044834 conn(0x7f7cc806d900 msgr2=0x7f7cc806dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:51.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 -- 172.21.15.77:0/3816044834 shutdown_connections 2023-07-04T18:51:51.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.862+0000 7f7ccec24700 1 -- 172.21.15.77:0/3816044834 wait complete. 2023-07-04T18:51:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.863+0000 7f7ccec24700 1 Processor -- start 2023-07-04T18:51:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.863+0000 7f7ccec24700 1 -- start start 2023-07-04T18:51:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 0x7f7cc81f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 0x7f7cc806c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81f2910 con 0x7f7cc80b2800 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80b4830 con 0x7f7cc80b0360 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccec24700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81f22c0 con 0x7f7cc80cd700 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44984/0 (socket says 172.21.15.77:44984) 2023-07-04T18:51:51.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3756397448 learned_addr learned my addr 172.21.15.77:0/3756397448 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccd1c1700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 0x7f7cc806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccc9c0700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 0x7f7cc81f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 msgr2=0x7f7cc806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 0x7f7cc806c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 msgr2=0x7f7cc81f1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 0x7f7cc81f1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:51.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7cc7fff700 1 -- 172.21.15.77:0/3756397448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb40094a0 con 0x7f7cc80b2800 2023-07-04T18:51:51.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccd1c1700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 0x7f7cc806c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:51.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.864+0000 7f7ccc9c0700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 0x7f7cc81f1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:51.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7cc7fff700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40090b0 tx=0x7f7cb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40045d0 con 0x7f7cc80b2800 2023-07-04T18:51:51.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb401f070 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc81f25a0 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40038c0 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.865+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc81f2f80 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.866+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc81f2a80 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.866+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7cb4003a20 con 0x7f7cc80b2800 2023-07-04T18:51:51.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.866+0000 7f7cc57fa700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 0x7f7cac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.866+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7cb4021070 con 0x7f7cc80b2800 2023-07-04T18:51:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.867+0000 7f7ccc9c0700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 0x7f7cac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:51.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.867+0000 7f7ccc9c0700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 0x7f7cac06d190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000f60 tx=0x7f7cbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:51.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:51.871+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7cb4055eb0 con 0x7f7cc80b2800 2023-07-04T18:51:52.060 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.056+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7cc804e210 con 0x7f7cc80b2800 2023-07-04T18:51:52.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.056+0000 7f7cc57fa700 1 -- 172.21.15.77:0/3756397448 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7cb40594d0 con 0x7f7cc80b2800 2023-07-04T18:51:52.061 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:52.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 msgr2=0x7f7cac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 0x7f7cac06d190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000f60 tx=0x7f7cbc009040 comp rx=0 tx=0).stop 2023-07-04T18:51:52.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 msgr2=0x7f7cc806bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40090b0 tx=0x7f7cb4004750 comp rx=0 tx=0).stop 2023-07-04T18:51:52.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 shutdown_connections 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7cac06ace0 0x7f7cac06d190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7cc80cd700 0x7f7cc806c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7cc80b0360 0x7f7cc81f1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 --2- 172.21.15.77:0/3756397448 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7cc80b2800 0x7f7cc806bce0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 >> 172.21.15.77:0/3756397448 conn(0x7f7cc806d900 msgr2=0x7f7cc80cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 shutdown_connections 2023-07-04T18:51:52.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.059+0000 7f7ccec24700 1 -- 172.21.15.77:0/3756397448 wait complete. 2023-07-04T18:51:52.072 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:52.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.194+0000 7f07b06b0700 1 Processor -- start 2023-07-04T18:51:52.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.194+0000 7f07b06b0700 1 -- start start 2023-07-04T18:51:52.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07b06b0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a80ce4c0 0x7f07a815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07b06b0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07a815d4a0 0x7f07a806fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07b06b0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a80ca0f0 con 0x7f07a815c760 2023-07-04T18:51:52.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07b06b0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a815cb30 con 0x7f07a815d4a0 2023-07-04T18:51:52.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07b06b0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a815dcd0 con 0x7f07a80ce4c0 2023-07-04T18:51:52.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07a815d4a0 0x7f07a806fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07ae44c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f07a815c760 0x7f07a80cddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52128/0 (socket says 172.21.15.77:52128) 2023-07-04T18:51:52.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07a815d4a0 0x7f07a806fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44994/0 (socket says 172.21.15.77:44994) 2023-07-04T18:51:52.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07ae44c700 1 -- 172.21.15.77:0/219473000 learned_addr learned my addr 172.21.15.77:0/219473000 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:52.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07adc4b700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a80ce4c0 0x7f07a815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 -- 172.21.15.77:0/219473000 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a80ce4c0 msgr2=0x7f07a815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:52.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a80ce4c0 0x7f07a815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 -- 172.21.15.77:0/219473000 >> v1:172.21.15.77:6789/0 conn(0x7f07a815c760 legacy=0x7f07a80cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07aec4d700 1 -- 172.21.15.77:0/219473000 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a80c97c0 con 0x7f07a815d4a0 2023-07-04T18:51:52.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.195+0000 7f07ad44a700 1 -- 172.21.15.77:0/219473000 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464980990 0 0) 0x7f07a80ca0f0 con 0x7f07a815c760 2023-07-04T18:51:52.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07aec4d700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a815d4a0 0x7f07a806fd70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009d90 tx=0x7f07a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705c0b6d1475d001 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07ad44a700 1 -- 172.21.15.77:0/219473000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a000e070 con 0x7f07a815d4a0 2023-07-04T18:51:52.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07ad44a700 1 -- 172.21.15.77:0/219473000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a000bbd0 con 0x7f07a815d4a0 2023-07-04T18:51:52.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07ad44a700 1 -- 172.21.15.77:0/219473000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0012430 con 0x7f07a815d4a0 2023-07-04T18:51:52.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 -- 172.21.15.77:0/219473000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a815d4a0 msgr2=0x7f07a806fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a815d4a0 0x7f07a806fd70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009d90 tx=0x7f07a000b330 comp rx=0 tx=0).stop 2023-07-04T18:51:52.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 -- 172.21.15.77:0/219473000 shutdown_connections 2023-07-04T18:51:52.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a815d4a0 0x7f07a806fd70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.206 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 --2- 172.21.15.77:0/219473000 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a80ce4c0 0x7f07a815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 -- 172.21.15.77:0/219473000 >> 172.21.15.77:0/219473000 conn(0x7f07a80cf0d0 msgr2=0x7f07a80cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 -- 172.21.15.77:0/219473000 shutdown_connections 2023-07-04T18:51:52.209 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.196+0000 7f07b06b0700 1 -- 172.21.15.77:0/219473000 wait complete. 2023-07-04T18:51:52.216 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.197+0000 7f07b06b0700 1 Processor -- start 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 -- start start 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 0x7f07a806a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 0x7f07a80c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81f4ec0 con 0x7f07a80ce4c0 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81f5110 con 0x7f07a815c760 2023-07-04T18:51:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07b06b0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81f53f0 con 0x7f07a815d4a0 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07ae44c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07ae44c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44996/0 (socket says 172.21.15.77:44996) 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.198+0000 7f07ae44c700 1 -- 172.21.15.77:0/263077935 learned_addr learned my addr 172.21.15.77:0/263077935 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07aec4d700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 0x7f07a80c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07adc4b700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 0x7f07a806a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 -- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 msgr2=0x7f07a80c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.218 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 0x7f07a80c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 -- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 msgr2=0x7f07a806a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 0x7f07a806a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 -- 172.21.15.77:0/263077935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0009a70 con 0x7f07a80ce4c0 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07aec4d700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 0x7f07a80c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07adc4b700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 0x7f07a806a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07ae44c700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07980011e0 tx=0x7f079800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798004c80 con 0x7f07a80ce4c0 2023-07-04T18:51:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798004e00 con 0x7f07a80ce4c0 2023-07-04T18:51:52.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a81f56d0 con 0x7f07a80ce4c0 2023-07-04T18:51:52.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798007310 con 0x7f07a80ce4c0 2023-07-04T18:51:52.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.199+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a81f5c90 con 0x7f07a80ce4c0 2023-07-04T18:51:52.220 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.200+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a80ca560 con 0x7f07a80ce4c0 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.200+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0798007470 con 0x7f07a80ce4c0 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.201+0000 7f079effd700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.201+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0798089220 con 0x7f07a80ce4c0 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.201+0000 7f07adc4b700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.201+0000 7f07adc4b700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 0x7f078c06d190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f07a80c91a0 tx=0x7f07a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.221 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.205+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f07a80ca560 con 0x7f07a80ce4c0 2023-07-04T18:51:52.394 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.392+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f07a80ca560 con 0x7f07a80ce4c0 2023-07-04T18:51:52.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.392+0000 7f079effd700 1 -- 172.21.15.77:0/263077935 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f07a80ca560 con 0x7f07a80ce4c0 2023-07-04T18:51:52.395 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:52.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 msgr2=0x7f078c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 0x7f078c06d190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f07a80c91a0 tx=0x7f07a4008040 comp rx=0 tx=0).stop 2023-07-04T18:51:52.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 msgr2=0x7f07a8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07980011e0 tx=0x7f079800cae0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 shutdown_connections 2023-07-04T18:51:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a80ce4c0 0x7f07a8069e90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f07a815d4a0 0x7f07a80c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 --2- 172.21.15.77:0/263077935 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07a815c760 0x7f07a806a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 >> 172.21.15.77:0/263077935 conn(0x7f07a80cf0d0 msgr2=0x7f07a80cddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 shutdown_connections 2023-07-04T18:51:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.394+0000 7f07b06b0700 1 -- 172.21.15.77:0/263077935 wait complete. 2023-07-04T18:51:52.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.531+0000 7f1524cd2700 1 Processor -- start 2023-07-04T18:51:52.533 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.531+0000 7f1524cd2700 1 -- start start 2023-07-04T18:51:52.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f1524cd2700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15201440b0 0x7f152015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f1524cd2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.540 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f1524cd2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152015f180 con 0x7f15200d0000 2023-07-04T18:51:52.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f1524cd2700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200d0480 con 0x7f15201440b0 2023-07-04T18:51:52.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f1524cd2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15200cee30 con 0x7f15200ce8c0 2023-07-04T18:51:52.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15201440b0 0x7f152015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f15200d0000 0x7f15200d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52144/0 (socket says 172.21.15.77:52144) 2023-07-04T18:51:52.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f15201440b0 0x7f152015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45004/0 (socket says 172.21.15.77:45004) 2023-07-04T18:51:52.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151ed9d700 1 -- 172.21.15.77:0/1061189583 learned_addr learned my addr 172.21.15.77:0/1061189583 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:52.542 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151dd9b700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 -- 172.21.15.77:0/1061189583 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 msgr2=0x7f15200cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 -- 172.21.15.77:0/1061189583 >> v1:172.21.15.77:6789/0 conn(0x7f15200d0000 legacy=0x7f15200d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 -- 172.21.15.77:0/1061189583 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15200cf110 con 0x7f15201440b0 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151d59a700 1 -- 172.21.15.77:0/1061189583 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644613417 0 0) 0x7f1518001490 con 0x7f15200d0000 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.532+0000 7f151e59c700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f152015ec40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1514009840 tx=0x7f1514009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a557005f507290d server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f151d59a700 1 -- 172.21.15.77:0/1061189583 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514011830 con 0x7f15201440b0 2023-07-04T18:51:52.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f151d59a700 1 -- 172.21.15.77:0/1061189583 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514011990 con 0x7f15201440b0 2023-07-04T18:51:52.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f151d59a700 1 -- 172.21.15.77:0/1061189583 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514011d20 con 0x7f15201440b0 2023-07-04T18:51:52.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 -- 172.21.15.77:0/1061189583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 msgr2=0x7f152015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f152015ec40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1514009840 tx=0x7f1514009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:52.544 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 -- 172.21.15.77:0/1061189583 shutdown_connections 2023-07-04T18:51:52.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f152015ec40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 --2- 172.21.15.77:0/1061189583 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 -- 172.21.15.77:0/1061189583 >> 172.21.15.77:0/1061189583 conn(0x7f152006d900 msgr2=0x7f152006dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 -- 172.21.15.77:0/1061189583 shutdown_connections 2023-07-04T18:51:52.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.533+0000 7f1524cd2700 1 -- 172.21.15.77:0/1061189583 wait complete. 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.534+0000 7f1524cd2700 1 Processor -- start 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 -- start start 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15200d0000 0x7f152015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15201f3210 con 0x7f15201440b0 2023-07-04T18:51:52.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b25a0 con 0x7f15200d0000 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f1524cd2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152015ce60 con 0x7f15200ce8c0 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f151ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f151ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45010/0 (socket says 172.21.15.77:45010) 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f151ed9d700 1 -- 172.21.15.77:0/2612789108 learned_addr learned my addr 172.21.15.77:0/2612789108 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f151e59c700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.535+0000 7f151dd9b700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15200d0000 0x7f152015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 -- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 msgr2=0x7f15200b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 -- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15200d0000 msgr2=0x7f152015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15200d0000 0x7f152015c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 -- 172.21.15.77:0/2612789108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15140094a0 con 0x7f15201440b0 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f151ed9d700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f151800cca0 tx=0x7f151800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151800ab70 con 0x7f15201440b0 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151800acd0 con 0x7f15201440b0 2023-07-04T18:51:52.548 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15201f2c50 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518007310 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.536+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15200c8f40 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.537+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1520067010 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.537+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1518007470 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.538+0000 7f150effd700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 0x7f150006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.538+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1518089040 con 0x7f15201440b0 2023-07-04T18:51:52.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.538+0000 7f151e59c700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 0x7f150006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.538+0000 7f151e59c700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 0x7f150006d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1514000c00 tx=0x7f1514008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.542+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1518054180 con 0x7f15201440b0 2023-07-04T18:51:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.726+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f152015f180 con 0x7f15201440b0 2023-07-04T18:51:52.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.726+0000 7f150effd700 1 -- 172.21.15.77:0/2612789108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1518018020 con 0x7f15201440b0 2023-07-04T18:51:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:52.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 msgr2=0x7f150006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 0x7f150006d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1514000c00 tx=0x7f1514008030 comp rx=0 tx=0).stop 2023-07-04T18:51:52.733 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 msgr2=0x7f15201f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f151800cca0 tx=0x7f151800a060 comp rx=0 tx=0).stop 2023-07-04T18:51:52.734 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 shutdown_connections 2023-07-04T18:51:52.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f150006ace0 0x7f150006d190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15201440b0 0x7f15201f2220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15200ce8c0 0x7f15200b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 --2- 172.21.15.77:0/2612789108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15200d0000 0x7f152015c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 >> 172.21.15.77:0/2612789108 conn(0x7f152006d900 msgr2=0x7f1520159fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 shutdown_connections 2023-07-04T18:51:52.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.728+0000 7f1524cd2700 1 -- 172.21.15.77:0/2612789108 wait complete. 2023-07-04T18:51:52.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:52.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.868+0000 7fbffa99d700 1 Processor -- start 2023-07-04T18:51:52.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 -- start start 2023-07-04T18:51:52.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff4069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbff40b2050 0x7fbff406a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbff4071110 0x7fbff4160060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff406d890 con 0x7fbff40b2050 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff4070f20 con 0x7fbff4071110 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbffa99d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff415c920 con 0x7fbff415c550 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.869+0000 7fbff919a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbff40b2050 0x7fbff406a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbff40b2050 0x7fbff406a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45018/0 (socket says 172.21.15.77:45018) 2023-07-04T18:51:52.873 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 -- 172.21.15.77:0/3886959420 learned_addr learned my addr 172.21.15.77:0/3886959420 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff999b700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff4069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbffa19c700 1 --2- 172.21.15.77:0/3886959420 >> v2:172.21.15.82:3300/0 conn(0x7fbff4071110 0x7fbff4160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 -- 172.21.15.77:0/3886959420 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 msgr2=0x7fbff4069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff4069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 -- 172.21.15.77:0/3886959420 >> v2:172.21.15.82:3300/0 conn(0x7fbff4071110 msgr2=0x7fbff4160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 --2- 172.21.15.77:0/3886959420 >> v2:172.21.15.82:3300/0 conn(0x7fbff4071110 0x7fbff4160060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 -- 172.21.15.77:0/3886959420 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4070960 con 0x7fbff40b2050 2023-07-04T18:51:52.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff999b700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff4069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff919a700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff40b2050 0x7fbff406a380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbff4161100 tx=0x7fbff0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881ab46c820e0404 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff8999700 1 -- 172.21.15.77:0/3886959420 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000b870 con 0x7fbff40b2050 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff8999700 1 -- 172.21.15.77:0/3886959420 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff000b9d0 con 0x7fbff40b2050 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.870+0000 7fbff8999700 1 -- 172.21.15.77:0/3886959420 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000bd60 con 0x7fbff40b2050 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 -- 172.21.15.77:0/3886959420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff40b2050 msgr2=0x7fbff406a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff40b2050 0x7fbff406a380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbff4161100 tx=0x7fbff0009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 -- 172.21.15.77:0/3886959420 shutdown_connections 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 --2- 172.21.15.77:0/3886959420 >> v2:172.21.15.82:3300/0 conn(0x7fbff4071110 0x7fbff4160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff4069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 --2- 172.21.15.77:0/3886959420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff40b2050 0x7fbff406a380 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 -- 172.21.15.77:0/3886959420 >> 172.21.15.77:0/3886959420 conn(0x7fbff406af50 msgr2=0x7fbff406b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 -- 172.21.15.77:0/3886959420 shutdown_connections 2023-07-04T18:51:52.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.871+0000 7fbffa99d700 1 -- 172.21.15.77:0/3886959420 wait complete. 2023-07-04T18:51:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.872+0000 7fbffa99d700 1 Processor -- start 2023-07-04T18:51:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.872+0000 7fbffa99d700 1 -- start start 2023-07-04T18:51:52.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff40b2050 0x7fbff40c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff40c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4160930 con 0x7fbff4071110 2023-07-04T18:51:52.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4158560 con 0x7fbff40b2050 2023-07-04T18:51:52.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4158840 con 0x7fbff415c550 2023-07-04T18:51:52.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45022/0 (socket says 172.21.15.77:45022) 2023-07-04T18:51:52.882 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 -- 172.21.15.77:0/2951689544 learned_addr learned my addr 172.21.15.77:0/2951689544 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:52.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa19c700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff40c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff919a700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff40b2050 0x7fbff40c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 -- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 msgr2=0x7fbff40c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.883 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff40c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 -- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff40b2050 msgr2=0x7fbff40c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:52.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff40b2050 0x7fbff40c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:52.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 -- 172.21.15.77:0/2951689544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff0009550 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbff999b700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c7f0 tx=0x7fbfe400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4005c20 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4005d80 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff4158b20 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4011820 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.873+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff41590e0 con 0x7fbff4071110 2023-07-04T18:51:52.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.874+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff404ea60 con 0x7fbff4071110 2023-07-04T18:51:52.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.875+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbfe4011980 con 0x7fbff4071110 2023-07-04T18:51:52.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.875+0000 7fbfea7fc700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 0x7fbfd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:52.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.875+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbfe4089270 con 0x7fbff4071110 2023-07-04T18:51:52.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.875+0000 7fbff919a700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 0x7fbfd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:52.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.876+0000 7fbff919a700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 0x7fbfd806d190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000c00 tx=0x7fbff0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:52.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:52.880+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbfe4057db0 con 0x7fbff4071110 2023-07-04T18:51:53.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.065+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbff40b1e30 con 0x7fbff4071110 2023-07-04T18:51:53.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.065+0000 7fbfea7fc700 1 -- 172.21.15.77:0/2951689544 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfe4057940 con 0x7fbff4071110 2023-07-04T18:51:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:53.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 msgr2=0x7fbfd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 0x7fbfd806d190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000c00 tx=0x7fbff0008030 comp rx=0 tx=0).stop 2023-07-04T18:51:53.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 msgr2=0x7fbff41603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400c7f0 tx=0x7fbfe400cbb0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 shutdown_connections 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbfd806ace0 0x7fbfd806d190 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbff4071110 0x7fbff41603f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbff415c550 0x7fbff40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 --2- 172.21.15.77:0/2951689544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff40b2050 0x7fbff40c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 >> 172.21.15.77:0/2951689544 conn(0x7fbff406af50 msgr2=0x7fbff415a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:53.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 shutdown_connections 2023-07-04T18:51:53.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.067+0000 7fbffa99d700 1 -- 172.21.15.77:0/2951689544 wait complete. 2023-07-04T18:51:53.116 INFO:tasks.cephfs_test_runner:Starting test: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-07-04T18:51:53.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic' 2023-07-04T18:51:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.237+0000 7fdf09edf700 1 Processor -- start 2023-07-04T18:51:53.239 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.237+0000 7fdf09edf700 1 -- start start 2023-07-04T18:51:53.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.238+0000 7fdf09edf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdf0406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf09edf700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406d890 con 0x7fdf040cdca0 2023-07-04T18:51:53.247 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf09edf700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406cc80 con 0x7fdf0406c770 2023-07-04T18:51:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf09edf700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0406cf60 con 0x7fdf04159280 2023-07-04T18:51:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf03fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf040cdca0 0x7fdf040ca010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52172/0 (socket says 172.21.15.77:52172) 2023-07-04T18:51:53.248 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf03fff700 1 -- 172.21.15.77:0/1981774917 learned_addr learned my addr 172.21.15.77:0/1981774917 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:53.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260550516 0 0) 0x7fdf0406d890 con 0x7fdf040cdca0 2023-07-04T18:51:53.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf037fe700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdf0406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.249 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec0057c0 con 0x7fdf040cdca0 2023-07-04T18:51:53.250 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347324552 0 0) 0x7fdeec0057c0 con 0x7fdf040cdca0 2023-07-04T18:51:53.250 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 msgr2=0x7fdf0406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:53.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdf0406c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 >> v1:172.21.15.82:6789/0 conn(0x7fdf0406c770 legacy=0x7fdf040cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0406d240 con 0x7fdf040cdca0 2023-07-04T18:51:53.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fdefc0019f0 con 0x7fdf040cdca0 2023-07-04T18:51:53.251 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 >> v1:172.21.15.77:6789/0 conn(0x7fdf040cdca0 legacy=0x7fdf040ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdeec005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdeec0061c0 0x7fdeec006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdeec007410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0406cf60 con 0x7fdeec006fa0 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec00dd90 con 0x7fdeec0061c0 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec00df50 con 0x7fdf04159280 2023-07-04T18:51:53.252 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf03fff700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdeec007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf037fe700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdeec005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.239+0000 7fdf02ffd700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdeec0061c0 0x7fdeec006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 -- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 msgr2=0x7fdeec005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf04159280 0x7fdeec005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 -- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdeec0061c0 msgr2=0x7fdeec006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdeec0061c0 0x7fdeec006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.253 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 -- 172.21.15.77:0/1981774917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0057c0 con 0x7fdeec006fa0 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdeec007410 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf040ce090 tx=0x7fdeec007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1791a26ed6381aea server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00a2f0 con 0x7fdeec006fa0 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc012620 con 0x7fdeec006fa0 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf027fc700 1 -- 172.21.15.77:0/1981774917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0129d0 con 0x7fdeec006fa0 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf09edf700 1 -- 172.21.15.77:0/1981774917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 msgr2=0x7fdeec007410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf09edf700 1 --2- 172.21.15.77:0/1981774917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdeec007410 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf040ce090 tx=0x7fdeec007a90 comp rx=0 tx=0).stop 2023-07-04T18:51:53.254 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf03fff700 1 -- 172.21.15.77:0/1981774917 reap_dead start 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf09edf700 1 -- 172.21.15.77:0/1981774917 shutdown_connections 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.240+0000 7fdf09edf700 1 -- 172.21.15.77:0/1981774917 >> 172.21.15.77:0/1981774917 conn(0x7fdf0406bba0 msgr2=0x7fdf040b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.241+0000 7fdf09edf700 1 -- 172.21.15.77:0/1981774917 shutdown_connections 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.241+0000 7fdf09edf700 1 -- 172.21.15.77:0/1981774917 wait complete. 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.242+0000 7fdf09edf700 1 Processor -- start 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.242+0000 7fdf09edf700 1 -- start start 2023-07-04T18:51:53.255 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 0x7fdf04153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf040ac8b0 0x7fdf040acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040c7330 con 0x7fdeec006fa0 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cd080 con 0x7fdf04153a70 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040cb300 con 0x7fdf040ac8b0 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.256 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45042/0 (socket says 172.21.15.77:45042) 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 -- 172.21.15.77:0/738176886 learned_addr learned my addr 172.21.15.77:0/738176886 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf03fff700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf040ac8b0 0x7fdf040acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf02ffd700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 0x7fdf04153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 -- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf040ac8b0 msgr2=0x7fdf040acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf040ac8b0 0x7fdf040acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 -- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 msgr2=0x7fdf04153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:53.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 0x7fdf04153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 -- 172.21.15.77:0/738176886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04154420 con 0x7fdeec006fa0 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf02ffd700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 0x7fdf04153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf037fe700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdef8011aa0 tx=0x7fdef8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef800f980 con 0x7fdeec006fa0 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef800fae0 con 0x7fdeec006fa0 2023-07-04T18:51:53.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef800fe90 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040c6a90 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.243+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040cb510 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.244+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0404e210 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.245+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdef8014440 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.245+0000 7fdeebfff700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 0x7fdef006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.245+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fdef8089c40 con 0x7fdeec006fa0 2023-07-04T18:51:53.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.245+0000 7fdf02ffd700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 0x7fdef006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.246+0000 7fdf02ffd700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 0x7fdef006d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf040b0b40 tx=0x7fdef4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:53.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.249+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdef8058840 con 0x7fdeec006fa0 2023-07-04T18:51:53.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:53.519+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]} v 0) v1 -- 0x7fdf040b1e80 con 0x7fdeec006fa0 2023-07-04T18:51:54.338 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.336+0000 7fdeebfff700 1 -- 172.21.15.77:0/738176886 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]}]=0 v53) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdef80220d0 con 0x7fdeec006fa0 2023-07-04T18:51:54.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 msgr2=0x7fdef006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 0x7fdef006d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf040b0b40 tx=0x7fdef4009040 comp rx=0 tx=0).stop 2023-07-04T18:51:54.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 msgr2=0x7fdf040cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdef8011aa0 tx=0x7fdef8011e60 comp rx=0 tx=0).stop 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 shutdown_connections 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.338+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdef006ace0 0x7fdef006d190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdeec006fa0 0x7fdf040cba30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf040ac8b0 0x7fdf040acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 --2- 172.21.15.77:0/738176886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf04153a70 0x7fdf04153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 >> 172.21.15.77:0/738176886 conn(0x7fdf0406bba0 msgr2=0x7fdf040b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 shutdown_connections 2023-07-04T18:51:54.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.339+0000 7fdf09edf700 1 -- 172.21.15.77:0/738176886 wait complete. 2023-07-04T18:51:54.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-07-04T18:51:54.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.478+0000 7f81ce968700 1 Processor -- start 2023-07-04T18:51:54.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 -- start start 2023-07-04T18:51:54.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c8069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 0x7f81c80b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c80ac8a0 con 0x7f81c806ea90 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c806a800 con 0x7f81c806a380 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.479+0000 7f81ce968700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c815d0b0 con 0x7f81c815d540 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c7fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f81c806ea90 0x7f81c8160a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52188/0 (socket says 172.21.15.77:52188) 2023-07-04T18:51:54.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c7fff700 1 -- 172.21.15.77:0/4176904579 learned_addr learned my addr 172.21.15.77:0/4176904579 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 --2- 172.21.15.77:0/4176904579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c8069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81ccf05700 1 --2- 172.21.15.77:0/4176904579 >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 0x7f81c80b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c6ffd700 1 -- 172.21.15.77:0/4176904579 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872227732 0 0) 0x7f81c80ac8a0 con 0x7f81c806ea90 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c6ffd700 1 -- 172.21.15.77:0/4176904579 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b40035b0 con 0x7f81c806ea90 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 -- 172.21.15.77:0/4176904579 >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 msgr2=0x7f81c80b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:54.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 --2- 172.21.15.77:0/4176904579 >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 0x7f81c80b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 -- 172.21.15.77:0/4176904579 >> v1:172.21.15.77:6789/0 conn(0x7f81c806ea90 legacy=0x7f81c8160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 -- 172.21.15.77:0/4176904579 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c815ca80 con 0x7f81c815d540 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81ccf05700 1 --2- 172.21.15.77:0/4176904579 >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 0x7f81c80b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c77fe700 1 --2- 172.21.15.77:0/4176904579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c8069e40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009840 tx=0x7f81b0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeeb50b07a536bba server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c6ffd700 1 -- 172.21.15.77:0/4176904579 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0011830 con 0x7f81c815d540 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c6ffd700 1 -- 172.21.15.77:0/4176904579 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b0011990 con 0x7f81c815d540 2023-07-04T18:51:54.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.480+0000 7f81c6ffd700 1 -- 172.21.15.77:0/4176904579 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0011d20 con 0x7f81c815d540 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 -- 172.21.15.77:0/4176904579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 msgr2=0x7f81c8069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 --2- 172.21.15.77:0/4176904579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c8069e40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009840 tx=0x7f81b0009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 -- 172.21.15.77:0/4176904579 shutdown_connections 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 --2- 172.21.15.77:0/4176904579 >> v2:172.21.15.82:3300/0 conn(0x7f81c806a380 0x7f81c80b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 --2- 172.21.15.77:0/4176904579 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c8069e40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 -- 172.21.15.77:0/4176904579 >> 172.21.15.77:0/4176904579 conn(0x7f81c806af50 msgr2=0x7f81c806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:54.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 -- 172.21.15.77:0/4176904579 shutdown_connections 2023-07-04T18:51:54.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.481+0000 7f81ce968700 1 -- 172.21.15.77:0/4176904579 wait complete. 2023-07-04T18:51:54.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.483+0000 7f81ce968700 1 Processor -- start 2023-07-04T18:51:54.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.483+0000 7f81ce968700 1 -- start start 2023-07-04T18:51:54.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81c806ea90 0x7f81c814cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c81f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81f5710 con 0x7f81c806a380 2023-07-04T18:51:54.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81f60e0 con 0x7f81c806ea90 2023-07-04T18:51:54.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8152d90 con 0x7f81c815d540 2023-07-04T18:51:54.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45056/0 (socket says 172.21.15.77:45056) 2023-07-04T18:51:54.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 -- 172.21.15.77:0/270455705 learned_addr learned my addr 172.21.15.77:0/270455705 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:54.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ccf05700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c81f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c77fe700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81c806ea90 0x7f81c814cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 -- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 msgr2=0x7f81c81f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c81f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 -- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81c806ea90 msgr2=0x7f81c814cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81c806ea90 0x7f81c814cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 -- 172.21.15.77:0/270455705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c0009a70 con 0x7f81c806a380 2023-07-04T18:51:54.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c7fff700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c700 tx=0x7f81b800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:54.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8005d70 con 0x7f81c806a380 2023-07-04T18:51:54.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b80070d0 con 0x7f81c806a380 2023-07-04T18:51:54.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b00094a0 con 0x7f81c806a380 2023-07-04T18:51:54.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.484+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8007460 con 0x7f81c806a380 2023-07-04T18:51:54.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.485+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c81f53a0 con 0x7f81c806a380 2023-07-04T18:51:54.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.485+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c81532b0 con 0x7f81c806a380 2023-07-04T18:51:54.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.486+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f81b80075c0 con 0x7f81c806a380 2023-07-04T18:51:54.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.486+0000 7f81c4ff9700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 0x7f81a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.486+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f81b80888c0 con 0x7f81c806a380 2023-07-04T18:51:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.486+0000 7f81c77fe700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 0x7f81a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.487+0000 7f81c77fe700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 0x7f81a806d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000c00 tx=0x7f81b001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:54.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.490+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f81c81532b0 con 0x7f81c806a380 2023-07-04T18:51:54.680 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.679+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f81c81f5200 con 0x7f81c806a380 2023-07-04T18:51:54.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.682+0000 7f81c4ff9700 1 -- 172.21.15.77:0/270455705 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f81c81f5200 con 0x7f81c806a380 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 msgr2=0x7f81a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 0x7f81a806d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000c00 tx=0x7f81b001c040 comp rx=0 tx=0).stop 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 msgr2=0x7f81c81527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c700 tx=0x7f81b800ca10 comp rx=0 tx=0).stop 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 shutdown_connections 2023-07-04T18:51:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f81a806ace0 0x7f81a806d190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81c806a380 0x7f81c81527b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f81c815d540 0x7f81c81f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 --2- 172.21.15.77:0/270455705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81c806ea90 0x7f81c814cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 >> 172.21.15.77:0/270455705 conn(0x7f81c806af50 msgr2=0x7f81c806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 shutdown_connections 2023-07-04T18:51:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:54.684+0000 7f81ce968700 1 -- 172.21.15.77:0/270455705 wait complete. 2023-07-04T18:51:54.698 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-07-04T18:51:54.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-07-04T18:51:54.754 DEBUG:teuthology.orchestra.run.smithi082:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-07-04T18:51:54.801 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-07-04T18:51:54.802 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:54.802 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:54.821 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:54.822 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:54.889 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:54.890 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:54.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:55.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.111+0000 7f02502a2700 1 Processor -- start 2023-07-04T18:51:55.113 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.111+0000 7f02502a2700 1 -- start start 2023-07-04T18:51:55.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f024815c640 0x7f02480cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f0248070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0248070db0 0x7f02480cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0248071250 con 0x7f0248070db0 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02480717f0 con 0x7f024815c640 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f02502a2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480cd290 con 0x7f024815ca10 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0248070db0 0x7f02480cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0248070db0 0x7f02480cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45068/0 (socket says 172.21.15.77:45068) 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 -- 172.21.15.77:0/369136210 learned_addr learned my addr 172.21.15.77:0/369136210 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024d83d700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f0248070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e03e700 1 --2- 172.21.15.77:0/369136210 >> v2:172.21.15.82:3300/0 conn(0x7f024815c640 0x7f02480cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 -- 172.21.15.77:0/369136210 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 msgr2=0x7f0248070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f0248070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 -- 172.21.15.77:0/369136210 >> v2:172.21.15.82:3300/0 conn(0x7f024815c640 msgr2=0x7f02480cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:55.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 --2- 172.21.15.77:0/369136210 >> v2:172.21.15.82:3300/0 conn(0x7f024815c640 0x7f02480cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 -- 172.21.15.77:0/369136210 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480ccfb0 con 0x7f0248070db0 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.112+0000 7f024e83f700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0248070db0 0x7f02480cc3d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0248069540 tx=0x7f024000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b008f6377b22eb server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f024d03c700 1 -- 172.21.15.77:0/369136210 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024000e070 con 0x7f0248070db0 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f024d03c700 1 -- 172.21.15.77:0/369136210 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024000b9c0 con 0x7f0248070db0 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f024d03c700 1 -- 172.21.15.77:0/369136210 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024000bd70 con 0x7f0248070db0 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 -- 172.21.15.77:0/369136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0248070db0 msgr2=0x7f02480cc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0248070db0 0x7f02480cc3d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0248069540 tx=0x7f024000b330 comp rx=0 tx=0).stop 2023-07-04T18:51:55.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 -- 172.21.15.77:0/369136210 shutdown_connections 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0248070db0 0x7f02480cc3d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 --2- 172.21.15.77:0/369136210 >> v2:172.21.15.82:3300/0 conn(0x7f024815c640 0x7f02480cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 --2- 172.21.15.77:0/369136210 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f0248070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 -- 172.21.15.77:0/369136210 >> 172.21.15.77:0/369136210 conn(0x7f024806bba0 msgr2=0x7f02480b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 -- 172.21.15.77:0/369136210 shutdown_connections 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.113+0000 7f02502a2700 1 -- 172.21.15.77:0/369136210 wait complete. 2023-07-04T18:51:55.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.114+0000 7f02502a2700 1 Processor -- start 2023-07-04T18:51:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.114+0000 7f02502a2700 1 -- start start 2023-07-04T18:51:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0248070db0 0x7f02481e4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f024815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480b2f10 con 0x7f024815c640 2023-07-04T18:51:55.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480b5560 con 0x7f0248070db0 2023-07-04T18:51:55.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f02502a2700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02481e53e0 con 0x7f024815ca10 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45072/0 (socket says 172.21.15.77:45072) 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 -- 172.21.15.77:0/2818419691 learned_addr learned my addr 172.21.15.77:0/2818419691 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024e83f700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f024815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024e03e700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0248070db0 0x7f02481e4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 -- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 msgr2=0x7f024815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f024815b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 -- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0248070db0 msgr2=0x7f02481e4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0248070db0 0x7f02481e4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024d83d700 1 -- 172.21.15.77:0/2818419691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009b30 con 0x7f024815c640 2023-07-04T18:51:55.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.115+0000 7f024e83f700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f024815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:55.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f024d83d700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f023800e7e0 tx=0x7f023800eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238013610 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238013770 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02480b4a10 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238013ae0 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.116+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480b4cc0 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.117+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024804e210 con 0x7f024815c640 2023-07-04T18:51:55.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.117+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0238013c40 con 0x7f024815c640 2023-07-04T18:51:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.117+0000 7f023effd700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 0x7f022c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.118+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0238089020 con 0x7f024815c640 2023-07-04T18:51:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.118+0000 7f024e03e700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 0x7f022c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.118+0000 7f024e03e700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 0x7f022c06d190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0244009770 tx=0x7f0244008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.122+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0238054160 con 0x7f024815c640 2023-07-04T18:51:55.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.310+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02480b2220 con 0x7f024815c640 2023-07-04T18:51:55.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.311+0000 7f023effd700 1 -- 172.21.15.77:0/2818419691 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f0238057780 con 0x7f024815c640 2023-07-04T18:51:55.313 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:55.314 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:55.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 msgr2=0x7f022c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 0x7f022c06d190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0244009770 tx=0x7f0244008040 comp rx=0 tx=0).stop 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 msgr2=0x7f02480c9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f023800e7e0 tx=0x7f023800eba0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 shutdown_connections 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f022c06ace0 0x7f022c06d190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f024815ca10 0x7f024815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0248070db0 0x7f02481e4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 --2- 172.21.15.77:0/2818419691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024815c640 0x7f02480c9350 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 >> 172.21.15.77:0/2818419691 conn(0x7f024806bba0 msgr2=0x7f02481e3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 shutdown_connections 2023-07-04T18:51:55.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.313+0000 7f02502a2700 1 -- 172.21.15.77:0/2818419691 wait complete. 2023-07-04T18:51:55.327 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:55.328 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:55.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.452+0000 7fd462293700 1 Processor -- start 2023-07-04T18:51:55.454 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.452+0000 7fd462293700 1 -- start start 2023-07-04T18:51:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd462293700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fd45c1590b0 0x7fd45c15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd462293700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd45c0b3330 0x7fd45c0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd462293700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c0b3f60 con 0x7fd45c06cfc0 2023-07-04T18:51:55.456 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd462293700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c06caa0 con 0x7fd45c0b3330 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd462293700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c15dc10 con 0x7fd45c1590b0 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd45c0b3330 0x7fd45c0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd45c0b3330 0x7fd45c0b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45078/0 (socket says 172.21.15.77:45078) 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4176221922 learned_addr learned my addr 172.21.15.77:0/4176221922 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45bfff700 1 --2- 172.21.15.77:0/4176221922 >> v2:172.21.15.82:3300/0 conn(0x7fd45c1590b0 0x7fd45c15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4176221922 >> v2:172.21.15.82:3300/0 conn(0x7fd45c1590b0 msgr2=0x7fd45c15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:55.457 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 --2- 172.21.15.77:0/4176221922 >> v2:172.21.15.82:3300/0 conn(0x7fd45c1590b0 0x7fd45c15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4176221922 >> v1:172.21.15.82:6789/0 conn(0x7fd45c06cfc0 legacy=0x7fd45c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4176221922 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c06c4e0 con 0x7fd45c0b3330 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45b7fe700 1 --2- 172.21.15.77:0/4176221922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c0b43f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0b26d0 tx=0x7fd44c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95039ed3ed54c9ad server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.453+0000 7fd45affd700 1 -- 172.21.15.77:0/4176221922 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c011830 con 0x7fd45c0b3330 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd45affd700 1 -- 172.21.15.77:0/4176221922 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c011990 con 0x7fd45c0b3330 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd45affd700 1 -- 172.21.15.77:0/4176221922 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c011d20 con 0x7fd45c0b3330 2023-07-04T18:51:55.458 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 -- 172.21.15.77:0/4176221922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 msgr2=0x7fd45c0b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 --2- 172.21.15.77:0/4176221922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c0b43f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0b26d0 tx=0x7fd44c009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 -- 172.21.15.77:0/4176221922 shutdown_connections 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 --2- 172.21.15.77:0/4176221922 >> v2:172.21.15.82:3300/0 conn(0x7fd45c1590b0 0x7fd45c15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 --2- 172.21.15.77:0/4176221922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c0b43f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 -- 172.21.15.77:0/4176221922 >> 172.21.15.77:0/4176221922 conn(0x7fd45c06af50 msgr2=0x7fd45c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 -- 172.21.15.77:0/4176221922 shutdown_connections 2023-07-04T18:51:55.459 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.454+0000 7fd462293700 1 -- 172.21.15.77:0/4176221922 wait complete. 2023-07-04T18:51:55.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.455+0000 7fd462293700 1 Processor -- start 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.455+0000 7fd462293700 1 -- start start 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 0x7fd45c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd45c1590b0 0x7fd45c0ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c1e5360 con 0x7fd45c0b3330 2023-07-04T18:51:55.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c1e4a90 con 0x7fd45c06cfc0 2023-07-04T18:51:55.464 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd462293700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c0ad6d0 con 0x7fd45c1590b0 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd45b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd45b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45084/0 (socket says 172.21.15.77:45084) 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4050000297 learned_addr learned my addr 172.21.15.77:0/4050000297 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd45bfff700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 0x7fd45c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.456+0000 7fd460830700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd45c1590b0 0x7fd45c0ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd45c1590b0 msgr2=0x7fd45c0ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.465 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd45c1590b0 0x7fd45c0ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 msgr2=0x7fd45c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 0x7fd45c157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 -- 172.21.15.77:0/4050000297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c009550 con 0x7fd45c0b3330 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45bfff700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 0x7fd45c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd45b7fe700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009c10 tx=0x7fd44c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c004530 con 0x7fd45c0b3330 2023-07-04T18:51:55.466 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44c01f070 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c1e4d10 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0038e0 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.457+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c15b390 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.458+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c1e5510 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.458+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd44c021030 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.459+0000 7fd458ff9700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 0x7fd44806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.459+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd44c08a9a0 con 0x7fd45c0b3330 2023-07-04T18:51:55.467 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.459+0000 7fd45bfff700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 0x7fd44806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.459+0000 7fd45bfff700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 0x7fd44806d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd45c157d50 tx=0x7fd450008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.463+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd45c1e5510 con 0x7fd45c0b3330 2023-07-04T18:51:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.651+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd45c04e210 con 0x7fd45c0b3330 2023-07-04T18:51:55.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.652+0000 7fd458ff9700 1 -- 172.21.15.77:0/4050000297 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd44c0150a0 con 0x7fd45c0b3330 2023-07-04T18:51:55.657 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:55.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 msgr2=0x7fd44806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 0x7fd44806d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd45c157d50 tx=0x7fd450008040 comp rx=0 tx=0).stop 2023-07-04T18:51:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 msgr2=0x7fd45c157640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009c10 tx=0x7fd44c004750 comp rx=0 tx=0).stop 2023-07-04T18:51:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 shutdown_connections 2023-07-04T18:51:55.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd44806ace0 0x7fd44806d190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd45c1590b0 0x7fd45c0ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd45c06cfc0 0x7fd45c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 --2- 172.21.15.77:0/4050000297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd45c0b3330 0x7fd45c157640 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 >> 172.21.15.77:0/4050000297 conn(0x7fd45c06af50 msgr2=0x7fd45c0b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 shutdown_connections 2023-07-04T18:51:55.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.654+0000 7fd462293700 1 -- 172.21.15.77:0/4050000297 wait complete. 2023-07-04T18:51:55.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:55.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.793+0000 7fa91aac6700 1 Processor -- start 2023-07-04T18:51:55.795 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.793+0000 7fa91aac6700 1 -- start start 2023-07-04T18:51:55.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa91aac6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa91415d540 0x7fa914069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.798 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa91aac6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140ac8a0 con 0x7fa91406a380 2023-07-04T18:51:55.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa91aac6700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91415d0b0 con 0x7fa91406ea90 2023-07-04T18:51:55.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa91aac6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91406a800 con 0x7fa91415d540 2023-07-04T18:51:55.799 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa913fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa91415d540 0x7fa914069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa919063700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa91406a380 0x7fa9140b2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52232/0 (socket says 172.21.15.77:52232) 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa919063700 1 -- 172.21.15.77:0/76100445 learned_addr learned my addr 172.21.15.77:0/76100445 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa9137fe700 1 -- 172.21.15.77:0/76100445 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680673529 0 0) 0x7fa9140ac8a0 con 0x7fa91406a380 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa9137fe700 1 -- 172.21.15.77:0/76100445 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9040035b0 con 0x7fa91406a380 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa913fff700 1 -- 172.21.15.77:0/76100445 >> v1:172.21.15.82:6789/0 conn(0x7fa91406ea90 legacy=0x7fa914160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.800 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa913fff700 1 -- 172.21.15.77:0/76100445 >> v1:172.21.15.77:6789/0 conn(0x7fa91406a380 legacy=0x7fa9140b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.794+0000 7fa913fff700 1 -- 172.21.15.77:0/76100445 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91415ca80 con 0x7fa91415d540 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa913fff700 1 --2- 172.21.15.77:0/76100445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91415d540 0x7fa914069e40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa900009840 tx=0x7fa900009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90517015e3e807f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa9137fe700 1 -- 172.21.15.77:0/76100445 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900011830 con 0x7fa91415d540 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa9137fe700 1 -- 172.21.15.77:0/76100445 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa900011990 con 0x7fa91415d540 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa9137fe700 1 -- 172.21.15.77:0/76100445 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900011d20 con 0x7fa91415d540 2023-07-04T18:51:55.801 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 -- 172.21.15.77:0/76100445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91415d540 msgr2=0x7fa914069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 --2- 172.21.15.77:0/76100445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91415d540 0x7fa914069e40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa900009840 tx=0x7fa900009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 -- 172.21.15.77:0/76100445 shutdown_connections 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 --2- 172.21.15.77:0/76100445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91415d540 0x7fa914069e40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 -- 172.21.15.77:0/76100445 >> 172.21.15.77:0/76100445 conn(0x7fa91406af50 msgr2=0x7fa91406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 -- 172.21.15.77:0/76100445 shutdown_connections 2023-07-04T18:51:55.802 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.795+0000 7fa91aac6700 1 -- 172.21.15.77:0/76100445 wait complete. 2023-07-04T18:51:55.803 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.796+0000 7fa91aac6700 1 Processor -- start 2023-07-04T18:51:55.804 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.796+0000 7fa91aac6700 1 -- start start 2023-07-04T18:51:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa91406ea90 0x7fa9141f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa91415d540 0x7fa9140ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141f5fd0 con 0x7fa91406a380 2023-07-04T18:51:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141f62b0 con 0x7fa91406ea90 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa91aac6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91415db60 con 0x7fa91415d540 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa918862700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa918862700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45096/0 (socket says 172.21.15.77:45096) 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.797+0000 7fa918862700 1 -- 172.21.15.77:0/1042353764 learned_addr learned my addr 172.21.15.77:0/1042353764 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 -- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa91415d540 msgr2=0x7fa9140ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa91415d540 0x7fa9140ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa913fff700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa91406ea90 0x7fa9141f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 -- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa91406ea90 msgr2=0x7fa9141f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa91406ea90 0x7fa9141f55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 -- 172.21.15.77:0/1042353764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9000094a0 con 0x7fa91406a380 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa918862700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c4e0 tx=0x7fa90800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080052d0 con 0x7fa91406a380 2023-07-04T18:51:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa908005430 con 0x7fa91406a380 2023-07-04T18:51:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91415de40 con 0x7fa91406a380 2023-07-04T18:51:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080038c0 con 0x7fa91406a380 2023-07-04T18:51:55.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.798+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9141f2050 con 0x7fa91406a380 2023-07-04T18:51:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.799+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa914073790 con 0x7fa91406a380 2023-07-04T18:51:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.799+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa908003a20 con 0x7fa91406a380 2023-07-04T18:51:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.800+0000 7fa9117fa700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 0x7fa8f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.800+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa908088990 con 0x7fa91406a380 2023-07-04T18:51:55.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.800+0000 7fa913fff700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 0x7fa8f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.800+0000 7fa913fff700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 0x7fa8f806d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9140cd4c0 tx=0x7fa900008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:55.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.804+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa908053a50 con 0x7fa91406a380 2023-07-04T18:51:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.990+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa91415d9c0 con 0x7fa91406a380 2023-07-04T18:51:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.990+0000 7fa9117fa700 1 -- 172.21.15.77:0/1042353764 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa91415d9c0 con 0x7fa91406a380 2023-07-04T18:51:55.992 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:55.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 msgr2=0x7fa8f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 0x7fa8f806d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9140cd4c0 tx=0x7fa900008030 comp rx=0 tx=0).stop 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 msgr2=0x7fa914151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c4e0 tx=0x7fa90800c8a0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 shutdown_connections 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa8f806ace0 0x7fa8f806d190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa91406a380 0x7fa914151580 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa91415d540 0x7fa9140ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 --2- 172.21.15.77:0/1042353764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa91406ea90 0x7fa9141f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 >> 172.21.15.77:0/1042353764 conn(0x7fa91406af50 msgr2=0x7fa9140af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 shutdown_connections 2023-07-04T18:51:55.996 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:55.992+0000 7fa91aac6700 1 -- 172.21.15.77:0/1042353764 wait complete. 2023-07-04T18:51:56.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.128+0000 7f69a6270700 1 Processor -- start 2023-07-04T18:51:56.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.128+0000 7f69a6270700 1 -- start start 2023-07-04T18:51:56.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a6270700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a006cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a6270700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a0159e60 con 0x7f69a006f4b0 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a6270700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a006b3d0 con 0x7f69a015bd80 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a6270700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a006b6b0 con 0x7f69a01598f0 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a480d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f69a006f4b0 0x7f69a00ce660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52246/0 (socket says 172.21.15.77:52246) 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.129+0000 7f69a480d700 1 -- 172.21.15.77:0/2060708213 learned_addr learned my addr 172.21.15.77:0/2060708213 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:56.133 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699f7fe700 1 --2- 172.21.15.77:0/2060708213 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a006cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862476961 0 0) 0x7f69a0159e60 con 0x7f69a006f4b0 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69880035b0 con 0x7f69a006f4b0 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228788670 0 0) 0x7f69a006b3d0 con 0x7f69a015bd80 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a0159e60 con 0x7f69a015bd80 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699f7fe700 1 -- 172.21.15.77:0/2060708213 >> v1:172.21.15.82:6789/0 conn(0x7f69a015bd80 legacy=0x7f69a015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699f7fe700 1 -- 172.21.15.77:0/2060708213 >> v1:172.21.15.77:6789/0 conn(0x7f69a006f4b0 legacy=0x7f69a00ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699f7fe700 1 -- 172.21.15.77:0/2060708213 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a006b990 con 0x7f69a01598f0 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699f7fe700 1 --2- 172.21.15.77:0/2060708213 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a006cfc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f699000aa10 tx=0x7f699000ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c8ba27ab4834e15 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000cb70 con 0x7f69a01598f0 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699000ccd0 con 0x7f69a01598f0 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.130+0000 7f699effd700 1 -- 172.21.15.77:0/2060708213 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990018600 con 0x7f69a01598f0 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 -- 172.21.15.77:0/2060708213 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 msgr2=0x7f69a006cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 --2- 172.21.15.77:0/2060708213 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a006cfc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f699000aa10 tx=0x7f699000ad20 comp rx=0 tx=0).stop 2023-07-04T18:51:56.135 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 -- 172.21.15.77:0/2060708213 shutdown_connections 2023-07-04T18:51:56.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 --2- 172.21.15.77:0/2060708213 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a006cfc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 -- 172.21.15.77:0/2060708213 >> 172.21.15.77:0/2060708213 conn(0x7f69a0065bd0 msgr2=0x7f69a0065fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:56.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 -- 172.21.15.77:0/2060708213 shutdown_connections 2023-07-04T18:51:56.136 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.131+0000 7f69a6270700 1 -- 172.21.15.77:0/2060708213 wait complete. 2023-07-04T18:51:56.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.132+0000 7f69a6270700 1 Processor -- start 2023-07-04T18:51:56.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.133+0000 7f69a6270700 1 -- start start 2023-07-04T18:51:56.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.133+0000 7f69a6270700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 0x7f69a0149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a014dc40 con 0x7f69a006f4b0 2023-07-04T18:51:56.141 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a014eaa0 con 0x7f69a015bd80 2023-07-04T18:51:56.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a014de90 con 0x7f69a01598f0 2023-07-04T18:51:56.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45110/0 (socket says 172.21.15.77:45110) 2023-07-04T18:51:56.142 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 -- 172.21.15.77:0/3651117833 learned_addr learned my addr 172.21.15.77:0/3651117833 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:56.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699f7fe700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a480d700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 0x7f69a0149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.143 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 -- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 msgr2=0x7f69a014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a014ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 -- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 msgr2=0x7f69a0149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 0x7f69a0149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.144 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 -- 172.21.15.77:0/3651117833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699000a640 con 0x7f69a006f4b0 2023-07-04T18:51:56.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a480d700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 0x7f69a0149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:56.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699f7fe700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:56.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699ffff700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f699400c550 tx=0x7f699400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.145 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940052d0 con 0x7f69a006f4b0 2023-07-04T18:51:56.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6994005430 con 0x7f69a006f4b0 2023-07-04T18:51:56.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a014e170 con 0x7f69a006f4b0 2023-07-04T18:51:56.146 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940038c0 con 0x7f69a006f4b0 2023-07-04T18:51:56.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.134+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a014e730 con 0x7f69a006f4b0 2023-07-04T18:51:56.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.135+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a014ed80 con 0x7f69a006f4b0 2023-07-04T18:51:56.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.136+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6994003a20 con 0x7f69a006f4b0 2023-07-04T18:51:56.147 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.136+0000 7f699cff9700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 0x7f698c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.136+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6994088a40 con 0x7f69a006f4b0 2023-07-04T18:51:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.136+0000 7f699f7fe700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 0x7f698c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.148 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.137+0000 7f699f7fe700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 0x7f698c06d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f699000a960 tx=0x7f6990012000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.149 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.140+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f699408ae60 con 0x7f69a006f4b0 2023-07-04T18:51:56.329 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.328+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69a014e8c0 con 0x7f69a006f4b0 2023-07-04T18:51:56.330 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.328+0000 7f699cff9700 1 -- 172.21.15.77:0/3651117833 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6994053af0 con 0x7f69a006f4b0 2023-07-04T18:51:56.330 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 msgr2=0x7f698c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 0x7f698c06d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f699000a960 tx=0x7f6990012000 comp rx=0 tx=0).stop 2023-07-04T18:51:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 msgr2=0x7f69a01487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f699400c550 tx=0x7f699400c910 comp rx=0 tx=0).stop 2023-07-04T18:51:56.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 shutdown_connections 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f698c06ace0 0x7f698c06d190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69a006f4b0 0x7f69a01487e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69a015bd80 0x7f69a0149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 --2- 172.21.15.77:0/3651117833 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f69a01598f0 0x7f69a014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 >> 172.21.15.77:0/3651117833 conn(0x7f69a0065bd0 msgr2=0x7f69a0158790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 shutdown_connections 2023-07-04T18:51:56.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.330+0000 7f69a6270700 1 -- 172.21.15.77:0/3651117833 wait complete. 2023-07-04T18:51:56.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:56.468 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 Processor -- start 2023-07-04T18:51:56.469 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 -- start start 2023-07-04T18:51:56.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 0x7f12bc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.471 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc0b3f60 con 0x7f12bc0b3330 2023-07-04T18:51:56.472 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc06caa0 con 0x7f12bc1590b0 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.467+0000 7f12c24ad700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0b4b00 con 0x7f12bc06cfc0 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 0x7f12bc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12bbfff700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12bb7fe700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f12bc0b3330 0x7f12bc0b43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60168/0 (socket says 172.21.15.77:60168) 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12bbfff700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:50924/0 (socket says 172.21.15.77:50924) 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12bb7fe700 1 -- 172.21.15.77:0/3826656587 learned_addr learned my addr 172.21.15.77:0/3826656587 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:56.473 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12baffd700 1 -- 172.21.15.77:0/3826656587 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960271025 0 0) 0x7f12bc0b3f60 con 0x7f12bc0b3330 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12baffd700 1 -- 172.21.15.77:0/3826656587 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a40035b0 con 0x7f12bc0b3330 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 -- 172.21.15.77:0/3826656587 >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 msgr2=0x7f12bc15d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 --2- 172.21.15.77:0/3826656587 >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 -- 172.21.15.77:0/3826656587 >> v1:172.21.15.82:6789/0 conn(0x7f12bc0b3330 legacy=0x7f12bc0b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 -- 172.21.15.77:0/3826656587 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc06d940 con 0x7f12bc06cfc0 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12bbfff700 1 --2- 172.21.15.77:0/3826656587 >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:56.474 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.468+0000 7f12c0a4a700 1 --2- 172.21.15.77:0/3826656587 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 0x7f12bc06d400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad1b29b2f03a106 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12baffd700 1 -- 172.21.15.77:0/3826656587 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400e070 con 0x7f12bc06cfc0 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12baffd700 1 -- 172.21.15.77:0/3826656587 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b400bb20 con 0x7f12bc06cfc0 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12baffd700 1 -- 172.21.15.77:0/3826656587 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4012430 con 0x7f12bc06cfc0 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 -- 172.21.15.77:0/3826656587 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 msgr2=0x7f12bc06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 --2- 172.21.15.77:0/3826656587 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 0x7f12bc06d400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b400b330 comp rx=0 tx=0).stop 2023-07-04T18:51:56.475 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 -- 172.21.15.77:0/3826656587 shutdown_connections 2023-07-04T18:51:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 --2- 172.21.15.77:0/3826656587 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc06cfc0 0x7f12bc06d400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 --2- 172.21.15.77:0/3826656587 >> v2:172.21.15.82:3300/0 conn(0x7f12bc1590b0 0x7f12bc15d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 -- 172.21.15.77:0/3826656587 >> 172.21.15.77:0/3826656587 conn(0x7f12bc06af50 msgr2=0x7f12bc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 -- 172.21.15.77:0/3826656587 shutdown_connections 2023-07-04T18:51:56.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.469+0000 7f12c24ad700 1 -- 172.21.15.77:0/3826656587 wait complete. 2023-07-04T18:51:56.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.470+0000 7f12c24ad700 1 Processor -- start 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 -- start start 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc0b3330 0x7f12bc0b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12bc1590b0 0x7f12bc15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0b04e0 con 0x7f12bc06cfc0 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0af860 con 0x7f12bc1590b0 2023-07-04T18:51:56.479 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.471+0000 7f12c24ad700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0afb40 con 0x7f12bc0b3330 2023-07-04T18:51:56.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45122/0 (socket says 172.21.15.77:45122) 2023-07-04T18:51:56.480 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 -- 172.21.15.77:0/178381226 learned_addr learned my addr 172.21.15.77:0/178381226 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bb7fe700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc0b3330 0x7f12bc0b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12c0a4a700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12bc1590b0 0x7f12bc15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 -- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc0b3330 msgr2=0x7f12bc0b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc0b3330 0x7f12bc0b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 -- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12bc1590b0 msgr2=0x7f12bc15a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12bc1590b0 0x7f12bc15a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 -- 172.21.15.77:0/178381226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac00b740 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12bbfff700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00ed40 tx=0x7f12ac00d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac00eee0 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac010640 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4009a70 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0109b0 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.472+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc0b0100 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.473+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc073790 con 0x7f12bc06cfc0 2023-07-04T18:51:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.474+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12ac010b10 con 0x7f12bc06cfc0 2023-07-04T18:51:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.474+0000 7f12b8ff9700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.474+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f12ac089df0 con 0x7f12bc06cfc0 2023-07-04T18:51:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.474+0000 7f12bb7fe700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.475+0000 7f12bb7fe700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 0x7f12a806d190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0adf10 tx=0x7f12b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.479+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12ac058930 con 0x7f12bc06cfc0 2023-07-04T18:51:56.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.665+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12bc04e210 con 0x7f12bc06cfc0 2023-07-04T18:51:56.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.665+0000 7f12b8ff9700 1 -- 172.21.15.77:0/178381226 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12ac0584c0 con 0x7f12bc06cfc0 2023-07-04T18:51:56.667 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 msgr2=0x7f12a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 0x7f12a806d190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f12bc0adf10 tx=0x7f12b0009040 comp rx=0 tx=0).stop 2023-07-04T18:51:56.669 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 msgr2=0x7f12bc0adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00ed40 tx=0x7f12ac00d630 comp rx=0 tx=0).stop 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 shutdown_connections 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f12a806ace0 0x7f12a806d190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12bc06cfc0 0x7f12bc0adf80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12bc1590b0 0x7f12bc15a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 --2- 172.21.15.77:0/178381226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f12bc0b3330 0x7f12bc0b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 >> 172.21.15.77:0/178381226 conn(0x7f12bc06af50 msgr2=0x7f12bc0b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 shutdown_connections 2023-07-04T18:51:56.671 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.667+0000 7f12c24ad700 1 -- 172.21.15.77:0/178381226 wait complete. 2023-07-04T18:51:56.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-04T18:51:56.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.801+0000 7fa30a392700 1 Processor -- start 2023-07-04T18:51:56.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.801+0000 7fa30a392700 1 -- start start 2023-07-04T18:51:56.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa30a392700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 0x7fa3040ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa30a392700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040ceeb0 con 0x7fa30406ee70 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa30a392700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040cd630 con 0x7fa3040d04e0 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa30a392700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30406ddd0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa3037fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa30406ee70 0x7fa30406d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52268/0 (socket says 172.21.15.77:52268) 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa3037fe700 1 -- 172.21.15.77:0/3617051672 learned_addr learned my addr 172.21.15.77:0/3617051672 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa303fff700 1 --2- 172.21.15.77:0/3617051672 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 0x7fa3040ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757954893 0 0) 0x7fa3040ceeb0 con 0x7fa30406ee70 2023-07-04T18:51:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec0035b0 con 0x7fa30406ee70 2023-07-04T18:51:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643562374 0 0) 0x7fa3040cd630 con 0x7fa3040d04e0 2023-07-04T18:51:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3040ceeb0 con 0x7fa3040d04e0 2023-07-04T18:51:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa303fff700 1 -- 172.21.15.77:0/3617051672 >> v1:172.21.15.82:6789/0 conn(0x7fa3040d04e0 legacy=0x7fa3040d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa303fff700 1 -- 172.21.15.77:0/3617051672 >> v1:172.21.15.77:6789/0 conn(0x7fa30406ee70 legacy=0x7fa30406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.802+0000 7fa303fff700 1 -- 172.21.15.77:0/3617051672 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30406e0b0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa303fff700 1 --2- 172.21.15.77:0/3617051672 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 0x7fa3040ce970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009850 tx=0x7fa2f8009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3574ef054a1da239 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8005e30 con 0x7fa3040ce5a0 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f8006030 con 0x7fa3040ce5a0 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa302ffd700 1 -- 172.21.15.77:0/3617051672 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80063a0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 -- 172.21.15.77:0/3617051672 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 msgr2=0x7fa3040ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 --2- 172.21.15.77:0/3617051672 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 0x7fa3040ce970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009850 tx=0x7fa2f8009b60 comp rx=0 tx=0).stop 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 -- 172.21.15.77:0/3617051672 shutdown_connections 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 --2- 172.21.15.77:0/3617051672 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040ce5a0 0x7fa3040ce970 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 -- 172.21.15.77:0/3617051672 >> 172.21.15.77:0/3617051672 conn(0x7fa3040664f0 msgr2=0x7fa304099d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 -- 172.21.15.77:0/3617051672 shutdown_connections 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.803+0000 7fa30a392700 1 -- 172.21.15.77:0/3617051672 wait complete. 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.805+0000 7fa30a392700 1 Processor -- start 2023-07-04T18:51:56.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.805+0000 7fa30a392700 1 -- start start 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.805+0000 7fa30a392700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa30406ee70 0x7fa304153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040d04e0 0x7fa3041e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30406b260 con 0x7fa3040ce5a0 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040ca8b0 con 0x7fa30406ee70 2023-07-04T18:51:56.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3040c62d0 con 0x7fa3040d04e0 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45138/0 (socket says 172.21.15.77:45138) 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 -- 172.21.15.77:0/4140259332 learned_addr learned my addr 172.21.15.77:0/4140259332 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30892f700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040d04e0 0x7fa3041e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 -- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040d04e0 msgr2=0x7fa3041e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040d04e0 0x7fa3041e4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa303fff700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa30406ee70 0x7fa304153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 -- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa30406ee70 msgr2=0x7fa304153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa30406ee70 0x7fa304153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 -- 172.21.15.77:0/4140259332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8009430 con 0x7fa3040ce5a0 2023-07-04T18:51:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa3037fe700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001eb0 tx=0x7fa2f400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40074e0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4007640 con 0x7fa3040ce5a0 2023-07-04T18:51:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3040c6550 con 0x7fa3040ce5a0 2023-07-04T18:51:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4005a70 con 0x7fa3040ce5a0 2023-07-04T18:51:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.806+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3040c6b10 con 0x7fa3040ce5a0 2023-07-04T18:51:56.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.807+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30404e210 con 0x7fa3040ce5a0 2023-07-04T18:51:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.808+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa2f4005bd0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.808+0000 7fa300ff9700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 0x7fa2f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.808+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa2f40889c0 con 0x7fa3040ce5a0 2023-07-04T18:51:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.808+0000 7fa303fff700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 0x7fa2f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:56.820 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.809+0000 7fa303fff700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 0x7fa2f006d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097a0 tx=0x7fa2f8008070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:56.821 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:56.812+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2f4053a70 con 0x7fa3040ce5a0 2023-07-04T18:51:57.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.012+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa3040718c0 con 0x7fa3040ce5a0 2023-07-04T18:51:57.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.013+0000 7fa300ff9700 1 -- 172.21.15.77:0/4140259332 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa2f4017080 con 0x7fa3040ce5a0 2023-07-04T18:51:57.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 msgr2=0x7fa2f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 0x7fa2f006d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80097a0 tx=0x7fa2f8008070 comp rx=0 tx=0).stop 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 msgr2=0x7fa3040c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4001eb0 tx=0x7fa2f400c860 comp rx=0 tx=0).stop 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 shutdown_connections 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa2f006ace0 0x7fa2f006d190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa3040d04e0 0x7fa3041e4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa30406ee70 0x7fa304153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 --2- 172.21.15.77:0/4140259332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3040ce5a0 0x7fa3040c5d90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 >> 172.21.15.77:0/4140259332 conn(0x7fa3040664f0 msgr2=0x7fa3040d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:57.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 shutdown_connections 2023-07-04T18:51:57.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.015+0000 7fa30a392700 1 -- 172.21.15.77:0/4140259332 wait complete. 2023-07-04T18:51:57.019 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2023-07-04T18:51:57.028 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-07-04T18:51:57.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:57.029 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-07-04T18:51:57.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:51:57.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:57.066 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-04T18:51:57.135 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:51:57.135 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:57.157 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:51:57.158 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:57.217 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:51:57.218 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:57.278 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:57.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.401+0000 7fad26046700 1 Processor -- start 2023-07-04T18:51:57.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.401+0000 7fad26046700 1 -- start start 2023-07-04T18:51:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad26046700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad2015ca10 0x7fad20070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad26046700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad26046700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad200cf830 con 0x7fad2015c640 2023-07-04T18:51:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad26046700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad200717f0 con 0x7fad2015ca10 2023-07-04T18:51:57.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad26046700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad200cd290 con 0x7fad20070db0 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad2015ca10 0x7fad20070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fad2015ca10 0x7fad20070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45146/0 (socket says 172.21.15.77:45146) 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 -- 172.21.15.77:0/1673658298 learned_addr learned my addr 172.21.15.77:0/1673658298 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1ffff700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 -- 172.21.15.77:0/1673658298 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 msgr2=0x7fad200cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 -- 172.21.15.77:0/1673658298 >> v1:172.21.15.82:6789/0 conn(0x7fad2015c640 legacy=0x7fad200cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 -- 172.21.15.77:0/1673658298 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad20071210 con 0x7fad2015ca10 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1ffff700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1effd700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad20070870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fad140098f0 tx=0x7fad14009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9550a1f76f951e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1e7fc700 1 -- 172.21.15.77:0/1673658298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14004230 con 0x7fad2015ca10 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1e7fc700 1 -- 172.21.15.77:0/1673658298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad14004390 con 0x7fad2015ca10 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.402+0000 7fad1e7fc700 1 -- 172.21.15.77:0/1673658298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140036a0 con 0x7fad2015ca10 2023-07-04T18:51:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 -- 172.21.15.77:0/1673658298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 msgr2=0x7fad20070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad20070870 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fad140098f0 tx=0x7fad14009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 -- 172.21.15.77:0/1673658298 shutdown_connections 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 --2- 172.21.15.77:0/1673658298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad20070870 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 -- 172.21.15.77:0/1673658298 >> 172.21.15.77:0/1673658298 conn(0x7fad2006bba0 msgr2=0x7fad200b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 -- 172.21.15.77:0/1673658298 shutdown_connections 2023-07-04T18:51:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.403+0000 7fad26046700 1 -- 172.21.15.77:0/1673658298 wait complete. 2023-07-04T18:51:57.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.404+0000 7fad26046700 1 Processor -- start 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.404+0000 7fad26046700 1 -- start start 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad2015c640 0x7fad200c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2015eda0 con 0x7fad2015ca10 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad200c6fd0 con 0x7fad2015c640 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad26046700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2014ad30 con 0x7fad20070db0 2023-07-04T18:51:57.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45154/0 (socket says 172.21.15.77:45154) 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 -- 172.21.15.77:0/2133461218 learned_addr learned my addr 172.21.15.77:0/2133461218 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1effd700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad2015c640 0x7fad200c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1f7fe700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 -- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 msgr2=0x7fad200c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 -- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad2015c640 msgr2=0x7fad200c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad2015c640 0x7fad200c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.405+0000 7fad1ffff700 1 -- 172.21.15.77:0/2133461218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14009550 con 0x7fad2015ca10 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad1ffff700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad1800cd80 tx=0x7fad1800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1800f480 con 0x7fad2015ca10 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1800f600 con 0x7fad2015ca10 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2015e910 con 0x7fad2015ca10 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1800f9b0 con 0x7fad2015ca10 2023-07-04T18:51:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.406+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2015e2c0 con 0x7fad2015ca10 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.407+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2015eaa0 con 0x7fad2015ca10 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.407+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fad1800b040 con 0x7fad2015ca10 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.408+0000 7fad07fff700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 0x7fad0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.408+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fad18089c90 con 0x7fad2015ca10 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.408+0000 7fad1f7fe700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 0x7fad0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.408+0000 7fad1f7fe700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 0x7fad0c06d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad10003e60 tx=0x7fad10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.412+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fad18054ca0 con 0x7fad2015ca10 2023-07-04T18:51:57.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.605+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad20070aa0 con 0x7fad2015ca10 2023-07-04T18:51:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.606+0000 7fad07fff700 1 -- 172.21.15.77:0/2133461218 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7fad180582c0 con 0x7fad2015ca10 2023-07-04T18:51:57.608 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:57.608 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 msgr2=0x7fad0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 0x7fad0c06d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad10003e60 tx=0x7fad10009040 comp rx=0 tx=0).stop 2023-07-04T18:51:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 msgr2=0x7fad2015acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad1800cd80 tx=0x7fad1800a560 comp rx=0 tx=0).stop 2023-07-04T18:51:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 shutdown_connections 2023-07-04T18:51:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad0c06ace0 0x7fad0c06d190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2015ca10 0x7fad2015acc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad20070db0 0x7fad200c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 --2- 172.21.15.77:0/2133461218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad2015c640 0x7fad200c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 >> 172.21.15.77:0/2133461218 conn(0x7fad2006bba0 msgr2=0x7fad20160320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 shutdown_connections 2023-07-04T18:51:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.608+0000 7fad26046700 1 -- 172.21.15.77:0/2133461218 wait complete. 2023-07-04T18:51:57.623 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:57.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:51:57.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.746+0000 7f4572918700 1 Processor -- start 2023-07-04T18:51:57.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.746+0000 7f4572918700 1 -- start start 2023-07-04T18:51:57.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4572918700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f456c06ea90 0x7f456c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4572918700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4572918700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c15da80 con 0x7f456c06a380 2023-07-04T18:51:57.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4572918700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456c06a800 con 0x7f456c06ea90 2023-07-04T18:51:57.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4572918700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0ac8a0 con 0x7f456c160a80 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456bfff700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f456c06ea90 0x7f456c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:35010/0 (socket says 172.21.15.77:35010) 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f4570eb5700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f456c06a380 0x7f456c0b2db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60210/0 (socket says 172.21.15.77:60210) 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 -- 172.21.15.77:0/797112525 learned_addr learned my addr 172.21.15.77:0/797112525 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 -- 172.21.15.77:0/797112525 >> v2:172.21.15.82:3300/0 conn(0x7f456c06ea90 msgr2=0x7f456c15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:57.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 --2- 172.21.15.77:0/797112525 >> v2:172.21.15.82:3300/0 conn(0x7f456c06ea90 0x7f456c15d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.747+0000 7f456b7fe700 1 -- 172.21.15.77:0/797112525 >> v1:172.21.15.82:6789/0 conn(0x7f456c06a380 legacy=0x7f456c0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f456b7fe700 1 -- 172.21.15.77:0/797112525 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c15d0b0 con 0x7f456c160a80 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f456b7fe700 1 --2- 172.21.15.77:0/797112525 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f455c0098f0 tx=0x7f455c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e76ae21e96a0c51 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f456affd700 1 -- 172.21.15.77:0/797112525 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c011830 con 0x7f456c160a80 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f456affd700 1 -- 172.21.15.77:0/797112525 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c011990 con 0x7f456c160a80 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f456affd700 1 -- 172.21.15.77:0/797112525 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c011d20 con 0x7f456c160a80 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 -- 172.21.15.77:0/797112525 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 msgr2=0x7f456c069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 --2- 172.21.15.77:0/797112525 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f455c0098f0 tx=0x7f455c009c00 comp rx=0 tx=0).stop 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 -- 172.21.15.77:0/797112525 shutdown_connections 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 --2- 172.21.15.77:0/797112525 >> v2:172.21.15.82:3300/0 conn(0x7f456c06ea90 0x7f456c15d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 --2- 172.21.15.77:0/797112525 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c069e40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.748+0000 7f4572918700 1 -- 172.21.15.77:0/797112525 >> 172.21.15.77:0/797112525 conn(0x7f456c06af50 msgr2=0x7f456c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.749+0000 7f4572918700 1 -- 172.21.15.77:0/797112525 shutdown_connections 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.749+0000 7f4572918700 1 -- 172.21.15.77:0/797112525 wait complete. 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.750+0000 7f4572918700 1 Processor -- start 2023-07-04T18:51:57.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.750+0000 7f4572918700 1 -- start start 2023-07-04T18:51:57.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f456c06a380 0x7f456c155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c0cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c1f51e0 con 0x7f456c06ea90 2023-07-04T18:51:57.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c155660 con 0x7f456c06a380 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0cd090 con 0x7f456c160a80 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45164/0 (socket says 172.21.15.77:45164) 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 -- 172.21.15.77:0/24707911 learned_addr learned my addr 172.21.15.77:0/24707911 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456bfff700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f456c06a380 0x7f456c155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4570eb5700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c0cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 -- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 msgr2=0x7f456c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c0cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 -- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f456c06a380 msgr2=0x7f456c155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f456c06a380 0x7f456c155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 -- 172.21.15.77:0/24707911 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c009550 con 0x7f456c06ea90 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f456b7fe700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f455c009c40 tx=0x7f455c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c0045d0 con 0x7f456c06ea90 2023-07-04T18:51:57.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c01e070 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c0cd370 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c0038e0 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.751+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c0cd930 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.752+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c04e210 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.754+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f455c003a40 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.754+0000 7f4568ff9700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 0x7f455806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.754+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f455c08a750 con 0x7f456c06ea90 2023-07-04T18:51:57.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.754+0000 7f456bfff700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 0x7f455806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:57.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.755+0000 7f456bfff700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 0x7f455806d190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f456c0cc1d0 tx=0x7f4560008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:57.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.758+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f455c055970 con 0x7f456c06ea90 2023-07-04T18:51:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.943+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f456c0d06f0 con 0x7f456c06ea90 2023-07-04T18:51:57.945 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.944+0000 7f4568ff9700 1 -- 172.21.15.77:0/24707911 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f455c058f90 con 0x7f456c06ea90 2023-07-04T18:51:57.946 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:51:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 msgr2=0x7f455806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 0x7f455806d190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f456c0cc1d0 tx=0x7f4560008040 comp rx=0 tx=0).stop 2023-07-04T18:51:57.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 msgr2=0x7f456c0cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f455c009c40 tx=0x7f455c004750 comp rx=0 tx=0).stop 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 shutdown_connections 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f455806ace0 0x7f455806d190 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f456c160a80 0x7f456c0cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f456c06a380 0x7f456c155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 --2- 172.21.15.77:0/24707911 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f456c06ea90 0x7f456c0cebc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:57.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 >> 172.21.15.77:0/24707911 conn(0x7f456c06af50 msgr2=0x7f456c0b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:57.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 shutdown_connections 2023-07-04T18:51:57.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:57.946+0000 7f4572918700 1 -- 172.21.15.77:0/24707911 wait complete. 2023-07-04T18:51:57.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:51:58.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.081+0000 7f1577eea700 1 Processor -- start 2023-07-04T18:51:58.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.081+0000 7f1577eea700 1 -- start start 2023-07-04T18:51:58.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1577eea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1570069e40 0x7f157006a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1577eea700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15700d01c0 0x7f15700ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1577eea700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15700cf750 con 0x7f15700ae970 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1577eea700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15700aedd0 con 0x7f1570069e40 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1577eea700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15700af060 con 0x7f15700d01c0 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1570069e40 0x7f157006a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1576487700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f15700ae970 0x7f157015db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52312/0 (socket says 172.21.15.77:52312) 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1576487700 1 -- 172.21.15.77:0/421778547 learned_addr learned my addr 172.21.15.77:0/421778547 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575485700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15700d01c0 0x7f15700ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 -- 172.21.15.77:0/421778547 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15700d01c0 msgr2=0x7f15700ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15700d01c0 0x7f15700ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 -- 172.21.15.77:0/421778547 >> v1:172.21.15.77:6789/0 conn(0x7f15700ae970 legacy=0x7f157015db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 -- 172.21.15.77:0/421778547 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700cf9c0 con 0x7f1570069e40 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1574c84700 1 -- 172.21.15.77:0/421778547 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287308127 0 0) 0x7f15700cf750 con 0x7f15700ae970 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1575c86700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1570069e40 0x7f157006a210 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f156c009840 tx=0x7f156c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c716e64144752e7e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.082+0000 7f1574c84700 1 -- 172.21.15.77:0/421778547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c011830 con 0x7f1570069e40 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1574c84700 1 -- 172.21.15.77:0/421778547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c011990 con 0x7f1570069e40 2023-07-04T18:51:58.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1574c84700 1 -- 172.21.15.77:0/421778547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c011d20 con 0x7f1570069e40 2023-07-04T18:51:58.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 -- 172.21.15.77:0/421778547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1570069e40 msgr2=0x7f157006a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1570069e40 0x7f157006a210 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f156c009840 tx=0x7f156c009b50 comp rx=0 tx=0).stop 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 -- 172.21.15.77:0/421778547 shutdown_connections 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1570069e40 0x7f157006a210 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 --2- 172.21.15.77:0/421778547 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f15700d01c0 0x7f15700ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 -- 172.21.15.77:0/421778547 >> 172.21.15.77:0/421778547 conn(0x7f157006af50 msgr2=0x7f157006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 -- 172.21.15.77:0/421778547 shutdown_connections 2023-07-04T18:51:58.091 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.083+0000 7f1577eea700 1 -- 172.21.15.77:0/421778547 wait complete. 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.084+0000 7f1577eea700 1 Processor -- start 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.084+0000 7f1577eea700 1 -- start start 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1570069e40 0x7f157015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 0x7f1570145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157015ac10 con 0x7f15700d01c0 2023-07-04T18:51:58.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15701584c0 con 0x7f15700ae970 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1577eea700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15701587a0 con 0x7f1570069e40 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1576487700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1576487700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45178/0 (socket says 172.21.15.77:45178) 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1576487700 1 -- 172.21.15.77:0/3364694845 learned_addr learned my addr 172.21.15.77:0/3364694845 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1575c86700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1570069e40 0x7f157015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.085+0000 7f1575485700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 0x7f1570145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 -- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1570069e40 msgr2=0x7f157015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1570069e40 0x7f157015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 -- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 msgr2=0x7f1570145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 0x7f1570145890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 -- 172.21.15.77:0/3364694845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c0094a0 con 0x7f15700d01c0 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1575485700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 0x7f1570145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1576487700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f156800cba0 tx=0x7f156800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156800aad0 con 0x7f15700d01c0 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156800ac50 con 0x7f15700d01c0 2023-07-04T18:51:58.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1570158a80 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15680070d0 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.086+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1570158190 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.087+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570156b30 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.087+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1568007230 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.088+0000 7f15667fc700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 0x7f155406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.088+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f1568089260 con 0x7f15700d01c0 2023-07-04T18:51:58.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.088+0000 7f1575c86700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 0x7f155406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.088+0000 7f1575c86700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 0x7f155406d190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f156c0090b0 tx=0x7f156c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.092+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1568054310 con 0x7f15700d01c0 2023-07-04T18:51:58.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.278+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f15700ad7d0 con 0x7f15700d01c0 2023-07-04T18:51:58.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.279+0000 7f15667fc700 1 -- 172.21.15.77:0/3364694845 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1568057930 con 0x7f15700d01c0 2023-07-04T18:51:58.281 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:51:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 msgr2=0x7f155406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 0x7f155406d190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f156c0090b0 tx=0x7f156c01c040 comp rx=0 tx=0).stop 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 msgr2=0x7f15701f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f156800cba0 tx=0x7f156800ceb0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 shutdown_connections 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f155406ace0 0x7f155406d190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f15700d01c0 0x7f15701f2740 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1570069e40 0x7f157015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 --2- 172.21.15.77:0/3364694845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15700ae970 0x7f1570145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 >> 172.21.15.77:0/3364694845 conn(0x7f157006af50 msgr2=0x7f15700af580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 shutdown_connections 2023-07-04T18:51:58.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.281+0000 7f1577eea700 1 -- 172.21.15.77:0/3364694845 wait complete. 2023-07-04T18:51:58.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:51:58.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.416+0000 7f32f2605700 1 Processor -- start 2023-07-04T18:51:58.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.416+0000 7f32f2605700 1 -- start start 2023-07-04T18:51:58.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f2605700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f32ec069e40 0x7f32ec06a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f2605700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec0cf750 con 0x7f32ec0aeb40 2023-07-04T18:51:58.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f2605700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec0af060 con 0x7f32ec0d01c0 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f2605700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ec0cf9c0 con 0x7f32ec069e40 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f0ba2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f32ec0aeb40 0x7f32ec15db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52324/0 (socket says 172.21.15.77:52324) 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32f0ba2700 1 -- 172.21.15.77:0/3568323326 learned_addr learned my addr 172.21.15.77:0/3568323326 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249401685 0 0) 0x7f32ec0cf750 con 0x7f32ec0aeb40 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32ebfff700 1 --2- 172.21.15.77:0/3568323326 >> v2:172.21.15.82:3300/0 conn(0x7f32ec069e40 0x7f32ec06a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d40035b0 con 0x7f32ec0aeb40 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 93380952 0 0) 0x7f32d40035b0 con 0x7f32ec0aeb40 2023-07-04T18:51:58.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 >> v2:172.21.15.82:3300/0 conn(0x7f32ec069e40 msgr2=0x7f32ec06a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 --2- 172.21.15.77:0/3568323326 >> v2:172.21.15.82:3300/0 conn(0x7f32ec069e40 0x7f32ec06a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 >> v1:172.21.15.82:6789/0 conn(0x7f32ec0d01c0 legacy=0x7f32ec0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec0cfca0 con 0x7f32ec0aeb40 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f32e40019f0 con 0x7f32ec0aeb40 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 >> v1:172.21.15.77:6789/0 conn(0x7f32ec0aeb40 legacy=0x7f32ec15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4003a00 0x7f32d4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 0x7f32d4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32d4005200 0x7f32d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec0cf9c0 con 0x7f32d4004850 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d400c080 con 0x7f32d4003a00 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.417+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d400c240 con 0x7f32d4005200 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 0x7f32d4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32ebfff700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4003a00 0x7f32d4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32f0ba2700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32d4005200 0x7f32d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 -- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32d4005200 msgr2=0x7f32d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32d4005200 0x7f32d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 -- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4003a00 msgr2=0x7f32d4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4003a00 0x7f32d4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32f0ba2700 1 -- 172.21.15.77:0/3568323326 reap_dead start 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 -- 172.21.15.77:0/3568323326 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40035b0 con 0x7f32d4004850 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eb7fe700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 0x7f32d4004cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0017c0 tx=0x7f32d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a5e69688f77609 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc0052d0 con 0x7f32d4004850 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc005430 con 0x7f32d4004850 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.418+0000 7f32eaffd700 1 -- 172.21.15.77:0/3568323326 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc0038c0 con 0x7f32d4004850 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 -- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 msgr2=0x7f32d4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 0x7f32d4004cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32dc0017c0 tx=0x7f32d4005880 comp rx=0 tx=0).stop 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 -- 172.21.15.77:0/3568323326 shutdown_connections 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 --2- 172.21.15.77:0/3568323326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32d4004850 0x7f32d4004cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 -- 172.21.15.77:0/3568323326 >> 172.21.15.77:0/3568323326 conn(0x7f32ec06af50 msgr2=0x7f32ec06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 -- 172.21.15.77:0/3568323326 shutdown_connections 2023-07-04T18:51:58.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.419+0000 7f32f2605700 1 -- 172.21.15.77:0/3568323326 wait complete. 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.420+0000 7f32f2605700 1 Processor -- start 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.420+0000 7f32f2605700 1 -- start start 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4004850 0x7f32ec154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32ec1f1e10 0x7f32ec15f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec153b10 con 0x7f32ec0b3c40 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f2390 con 0x7f32d4004850 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f2605700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1f2670 con 0x7f32ec1f1e10 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45198/0 (socket says 172.21.15.77:45198) 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 -- 172.21.15.77:0/2421678622 learned_addr learned my addr 172.21.15.77:0/2421678622 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32eb7fe700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32ec1f1e10 0x7f32ec15f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32ebfff700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4004850 0x7f32ec154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 -- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32ec1f1e10 msgr2=0x7f32ec15f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32ec1f1e10 0x7f32ec15f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 -- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4004850 msgr2=0x7f32ec154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4004850 0x7f32ec154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 -- 172.21.15.77:0/2421678622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec1f2950 con 0x7f32ec0b3c40 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32f0ba2700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000f80 tx=0x7f32e400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.421+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e400ad60 con 0x7f32ec0b3c40 2023-07-04T18:51:58.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.422+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e40133f0 con 0x7f32ec0b3c40 2023-07-04T18:51:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.422+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec1f2c30 con 0x7f32ec0b3c40 2023-07-04T18:51:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.422+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e40137a0 con 0x7f32ec0b3c40 2023-07-04T18:51:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.422+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec15fbc0 con 0x7f32ec0b3c40 2023-07-04T18:51:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.422+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec04e210 con 0x7f32ec0b3c40 2023-07-04T18:51:58.431 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.423+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f32e4013900 con 0x7f32ec0b3c40 2023-07-04T18:51:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.423+0000 7f32e8ff9700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 0x7f32d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.423+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f32e4089c20 con 0x7f32ec0b3c40 2023-07-04T18:51:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.424+0000 7f32ebfff700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 0x7f32d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.424+0000 7f32ebfff700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 0x7f32d806d190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009c60 tx=0x7f32e0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.428+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f32e4054cd0 con 0x7f32ec0b3c40 2023-07-04T18:51:58.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.611+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32ec071570 con 0x7f32ec0b3c40 2023-07-04T18:51:58.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.611+0000 7f32e8ff9700 1 -- 172.21.15.77:0/2421678622 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32e40582f0 con 0x7f32ec0b3c40 2023-07-04T18:51:58.614 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:51:58.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 msgr2=0x7f32d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 0x7f32d806d190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009c60 tx=0x7f32e0009210 comp rx=0 tx=0).stop 2023-07-04T18:51:58.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 msgr2=0x7f32ec141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000f80 tx=0x7f32e400a5f0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 shutdown_connections 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f32d806ace0 0x7f32d806d190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32ec0b3c40 0x7f32ec141090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32d4004850 0x7f32ec154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 --2- 172.21.15.77:0/2421678622 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f32ec1f1e10 0x7f32ec15f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.613+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 >> 172.21.15.77:0/2421678622 conn(0x7f32ec06af50 msgr2=0x7f32ec160000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.614+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 shutdown_connections 2023-07-04T18:51:58.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.614+0000 7f32f2605700 1 -- 172.21.15.77:0/2421678622 wait complete. 2023-07-04T18:51:58.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:51:58.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.748+0000 7fcd8322d700 1 Processor -- start 2023-07-04T18:51:58.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 -- start start 2023-07-04T18:51:58.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 0x7fcd7c15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c15f180 con 0x7fcd7c0d0000 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c0d0440 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.749+0000 7fcd8322d700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c0cee30 con 0x7fcd7c1440b0 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45204/0 (socket says 172.21.15.77:45204) 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 -- 172.21.15.77:0/10735387 learned_addr learned my addr 172.21.15.77:0/10735387 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:58.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/10735387 >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 0x7fcd7c15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 -- 172.21.15.77:0/10735387 >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 msgr2=0x7fcd7c15ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 --2- 172.21.15.77:0/10735387 >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 0x7fcd7c15ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 -- 172.21.15.77:0/10735387 >> v1:172.21.15.82:6789/0 conn(0x7fcd7c0d0000 legacy=0x7fcd7c0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:58.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 -- 172.21.15.77:0/10735387 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c0cf110 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/10735387 >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 0x7fcd7c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:51:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7bfff700 1 --2- 172.21.15.77:0/10735387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0669f0 tx=0x7fcd68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5782f1f94e0ead server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7b7fe700 1 -- 172.21.15.77:0/10735387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68004230 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7b7fe700 1 -- 172.21.15.77:0/10735387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68004390 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.750+0000 7fcd7b7fe700 1 -- 172.21.15.77:0/10735387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd680036a0 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 -- 172.21.15.77:0/10735387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0ce8c0 msgr2=0x7fcd7c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 --2- 172.21.15.77:0/10735387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0669f0 tx=0x7fcd68009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 -- 172.21.15.77:0/10735387 shutdown_connections 2023-07-04T18:51:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 --2- 172.21.15.77:0/10735387 >> v2:172.21.15.82:3300/0 conn(0x7fcd7c1440b0 0x7fcd7c15ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 --2- 172.21.15.77:0/10735387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c0cfac0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 -- 172.21.15.77:0/10735387 >> 172.21.15.77:0/10735387 conn(0x7fcd7c06d900 msgr2=0x7fcd7c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 -- 172.21.15.77:0/10735387 shutdown_connections 2023-07-04T18:51:58.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.751+0000 7fcd8322d700 1 -- 172.21.15.77:0/10735387 wait complete. 2023-07-04T18:51:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.752+0000 7fcd8322d700 1 Processor -- start 2023-07-04T18:51:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.752+0000 7fcd8322d700 1 -- start start 2023-07-04T18:51:58.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c1efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcd7c1440b0 0x7fcd7c15ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0b4cb0 con 0x7fcd7c0d0000 2023-07-04T18:51:58.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0c6350 con 0x7fcd7c0ce8c0 2023-07-04T18:51:58.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd8322d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0c6630 con 0x7fcd7c1440b0 2023-07-04T18:51:58.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45208/0 (socket says 172.21.15.77:45208) 2023-07-04T18:51:58.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 -- 172.21.15.77:0/2237275797 learned_addr learned my addr 172.21.15.77:0/2237275797 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:58.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd817ca700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcd7c1440b0 0x7fcd7c15ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c1efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 -- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcd7c1440b0 msgr2=0x7fcd7c15ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcd7c1440b0 0x7fcd7c15ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 -- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 msgr2=0x7fcd7c1efbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c1efbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 -- 172.21.15.77:0/2237275797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68009550 con 0x7fcd7c0d0000 2023-07-04T18:51:58.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c1efbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:51:58.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.753+0000 7fcd7bfff700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0d1df0 tx=0x7fcd68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.754+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68003cd0 con 0x7fcd7c0d0000 2023-07-04T18:51:58.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.754+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6801f070 con 0x7fcd7c0d0000 2023-07-04T18:51:58.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.754+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c0c6910 con 0x7fcd7c0d0000 2023-07-04T18:51:58.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.754+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68011870 con 0x7fcd7c0d0000 2023-07-04T18:51:58.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.754+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c0c6ed0 con 0x7fcd7c0d0000 2023-07-04T18:51:58.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.755+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c04e210 con 0x7fcd7c0d0000 2023-07-04T18:51:58.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.755+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcd68021030 con 0x7fcd7c0d0000 2023-07-04T18:51:58.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.755+0000 7fcd797fa700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:58.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.756+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fcd6808a8b0 con 0x7fcd7c0d0000 2023-07-04T18:51:58.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.756+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:58.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.756+0000 7fcd80fc9700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 0x7fcd6006d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000aa20 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:58.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.760+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcd680559f0 con 0x7fcd7c0d0000 2023-07-04T18:51:58.946 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.944+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd7c15f180 con 0x7fcd7c0d0000 2023-07-04T18:51:58.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.945+0000 7fcd797fa700 1 -- 172.21.15.77:0/2237275797 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd7c15f180 con 0x7fcd7c0d0000 2023-07-04T18:51:58.947 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:51:58.948 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 msgr2=0x7fcd6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 0x7fcd6006d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000aa20 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2023-07-04T18:51:58.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 msgr2=0x7fcd7c15c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:58.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0d1df0 tx=0x7fcd68004750 comp rx=0 tx=0).stop 2023-07-04T18:51:58.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 shutdown_connections 2023-07-04T18:51:58.949 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcd6006ace0 0x7fcd6006d190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcd7c1440b0 0x7fcd7c15ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd7c0ce8c0 0x7fcd7c1efbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 --2- 172.21.15.77:0/2237275797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd7c0d0000 0x7fcd7c15c950 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:58.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 >> 172.21.15.77:0/2237275797 conn(0x7fcd7c06d900 msgr2=0x7fcd7c158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:58.950 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 shutdown_connections 2023-07-04T18:51:58.951 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:58.947+0000 7fcd8322d700 1 -- 172.21.15.77:0/2237275797 wait complete. 2023-07-04T18:51:58.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-04T18:51:59.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.080+0000 7fe83c8b4700 1 Processor -- start 2023-07-04T18:51:59.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.080+0000 7fe83c8b4700 1 -- start start 2023-07-04T18:51:59.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815a430 0x7fe8380ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8380cebf0 0x7fe8380cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fe83815ea40 0x7fe8380cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8380cfe60 con 0x7fe8380cebf0 2023-07-04T18:51:59.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8380cf100 con 0x7fe83815ea40 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83c8b4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83815df60 con 0x7fe83815a430 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8380cebf0 0x7fe8380cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe8380cebf0 0x7fe8380cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45216/0 (socket says 172.21.15.77:45216) 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 -- 172.21.15.77:0/1930165918 learned_addr learned my addr 172.21.15.77:0/1930165918 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83659c700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815a430 0x7fe8380ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe836d9d700 1 --2- 172.21.15.77:0/1930165918 >> v2:172.21.15.82:3300/0 conn(0x7fe83815ea40 0x7fe8380cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 -- 172.21.15.77:0/1930165918 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815a430 msgr2=0x7fe8380ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815a430 0x7fe8380ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 -- 172.21.15.77:0/1930165918 >> v2:172.21.15.82:3300/0 conn(0x7fe83815ea40 msgr2=0x7fe8380cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 --2- 172.21.15.77:0/1930165918 >> v2:172.21.15.82:3300/0 conn(0x7fe83815ea40 0x7fe8380cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 -- 172.21.15.77:0/1930165918 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83815e240 con 0x7fe8380cebf0 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe835d9b700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380cebf0 0x7fe8380cf3e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe838066b50 tx=0x7fe82c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b844350a2c82cd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83559a700 1 -- 172.21.15.77:0/1930165918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c011830 con 0x7fe8380cebf0 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.081+0000 7fe83559a700 1 -- 172.21.15.77:0/1930165918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c011990 con 0x7fe8380cebf0 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83559a700 1 -- 172.21.15.77:0/1930165918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c011d20 con 0x7fe8380cebf0 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 -- 172.21.15.77:0/1930165918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380cebf0 msgr2=0x7fe8380cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380cebf0 0x7fe8380cf3e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe838066b50 tx=0x7fe82c009bd0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 -- 172.21.15.77:0/1930165918 shutdown_connections 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/1930165918 >> v2:172.21.15.82:3300/0 conn(0x7fe83815ea40 0x7fe8380cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815a430 0x7fe8380ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/1930165918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380cebf0 0x7fe8380cf3e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 -- 172.21.15.77:0/1930165918 >> 172.21.15.77:0/1930165918 conn(0x7fe83806d7c0 msgr2=0x7fe83806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 -- 172.21.15.77:0/1930165918 shutdown_connections 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.082+0000 7fe83c8b4700 1 -- 172.21.15.77:0/1930165918 wait complete. 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.084+0000 7fe83c8b4700 1 Processor -- start 2023-07-04T18:51:59.090 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.084+0000 7fe83c8b4700 1 -- start start 2023-07-04T18:51:59.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8380cebf0 0x7fe838149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815ea40 0x7fe8380c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83814a4c0 con 0x7fe8380ca810 2023-07-04T18:51:59.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381543f0 con 0x7fe8380cebf0 2023-07-04T18:51:59.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83c8b4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8380c9770 con 0x7fe83815ea40 2023-07-04T18:51:59.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45224/0 (socket says 172.21.15.77:45224) 2023-07-04T18:51:59.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 -- 172.21.15.77:0/2300008307 learned_addr learned my addr 172.21.15.77:0/2300008307 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:59.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe835d9b700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815ea40 0x7fe8380c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe83659c700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8380cebf0 0x7fe838149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 -- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815ea40 msgr2=0x7fe8380c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815ea40 0x7fe8380c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 -- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8380cebf0 msgr2=0x7fe838149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8380cebf0 0x7fe838149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 -- 172.21.15.77:0/2300008307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c009550 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.085+0000 7fe836d9d700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe83000efc0 tx=0x7fe83000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.086+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000ca70 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.086+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83000f040 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.086+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8380c9a50 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.086+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000ce10 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.086+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8380ca010 con 0x7fe8380ca810 2023-07-04T18:51:59.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.087+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838067050 con 0x7fe8380ca810 2023-07-04T18:51:59.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.087+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe830014480 con 0x7fe8380ca810 2023-07-04T18:51:59.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.087+0000 7fe826ffd700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 0x7fe81806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.088+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe830089b60 con 0x7fe8380ca810 2023-07-04T18:51:59.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.088+0000 7fe83659c700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 0x7fe81806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.088+0000 7fe83659c700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 0x7fe81806d190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe828009be0 tx=0x7fe828009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.098 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.092+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe830054c10 con 0x7fe8380ca810 2023-07-04T18:51:59.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.292+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe83804e210 con 0x7fe8380ca810 2023-07-04T18:51:59.294 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.293+0000 7fe826ffd700 1 -- 172.21.15.77:0/2300008307 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe830058230 con 0x7fe8380ca810 2023-07-04T18:51:59.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 msgr2=0x7fe81806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.301 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 0x7fe81806d190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe828009be0 tx=0x7fe828009210 comp rx=0 tx=0).stop 2023-07-04T18:51:59.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 msgr2=0x7fe8380c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe83000efc0 tx=0x7fe83000c0c0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 shutdown_connections 2023-07-04T18:51:59.302 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe81806ace0 0x7fe81806d190 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8380ca810 0x7fe8380c9140 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8380cebf0 0x7fe838149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 --2- 172.21.15.77:0/2300008307 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe83815ea40 0x7fe8380c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.303 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 >> 172.21.15.77:0/2300008307 conn(0x7fe83806d7c0 msgr2=0x7fe83815c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:59.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 shutdown_connections 2023-07-04T18:51:59.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.295+0000 7fe83c8b4700 1 -- 172.21.15.77:0/2300008307 wait complete. 2023-07-04T18:51:59.304 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2023-07-04T18:51:59.308 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-07-04T18:51:59.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:59.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-07-04T18:51:59.351 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:51:59.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:51:59.352 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-07-04T18:51:59.420 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:59.421 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:59.438 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:59.438 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:59.495 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:51:59.495 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:51:59.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:51:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.714+0000 7f8bebe61700 1 Processor -- start 2023-07-04T18:51:59.715 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.714+0000 7f8bebe61700 1 -- start start 2023-07-04T18:51:59.716 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bebe61700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 0x7f8be4160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.716 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bebe61700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bebe61700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be40b0270 con 0x7f8be406ef20 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bebe61700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be40d02e0 con 0x7f8be415c510 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bebe61700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be415c8e0 con 0x7f8be40cf960 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8be9bfd700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 0x7f8be4160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bea3fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8be93fc700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f8be406ef20 0x7f8be40cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60278/0 (socket says 172.21.15.77:60278) 2023-07-04T18:51:59.717 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8be93fc700 1 -- 172.21.15.77:0/2420650110 learned_addr learned my addr 172.21.15.77:0/2420650110 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:51:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8be8bfb700 1 -- 172.21.15.77:0/2420650110 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883527794 0 0) 0x7f8be40b0270 con 0x7f8be406ef20 2023-07-04T18:51:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8be8bfb700 1 -- 172.21.15.77:0/2420650110 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd00035b0 con 0x7f8be406ef20 2023-07-04T18:51:59.719 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bea3fe700 1 -- 172.21.15.77:0/2420650110 >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 msgr2=0x7f8be4160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.715+0000 7f8bea3fe700 1 --2- 172.21.15.77:0/2420650110 >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 0x7f8be4160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bea3fe700 1 -- 172.21.15.77:0/2420650110 >> v1:172.21.15.82:6789/0 conn(0x7f8be406ef20 legacy=0x7f8be40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bea3fe700 1 -- 172.21.15.77:0/2420650110 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4160f90 con 0x7f8be40cf960 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8be9bfd700 1 --2- 172.21.15.77:0/2420650110 >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 0x7f8be4160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bea3fe700 1 --2- 172.21.15.77:0/2420650110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40cfda0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc009d90 tx=0x7f8bdc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d74c0fdc281dd892 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8be8bfb700 1 -- 172.21.15.77:0/2420650110 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc00e070 con 0x7f8be40cf960 2023-07-04T18:51:59.720 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8be8bfb700 1 -- 172.21.15.77:0/2420650110 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc00bb20 con 0x7f8be40cf960 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8be8bfb700 1 -- 172.21.15.77:0/2420650110 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc00be90 con 0x7f8be40cf960 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bebe61700 1 -- 172.21.15.77:0/2420650110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 msgr2=0x7f8be40cfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bebe61700 1 --2- 172.21.15.77:0/2420650110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40cfda0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc009d90 tx=0x7f8bdc00b330 comp rx=0 tx=0).stop 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bebe61700 1 -- 172.21.15.77:0/2420650110 shutdown_connections 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bebe61700 1 --2- 172.21.15.77:0/2420650110 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40cfda0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.716+0000 7f8bebe61700 1 --2- 172.21.15.77:0/2420650110 >> v2:172.21.15.82:3300/0 conn(0x7f8be415c510 0x7f8be4160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.717+0000 7f8bebe61700 1 -- 172.21.15.77:0/2420650110 >> 172.21.15.77:0/2420650110 conn(0x7f8be406d7c0 msgr2=0x7f8be406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:59.721 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.717+0000 7f8bebe61700 1 -- 172.21.15.77:0/2420650110 shutdown_connections 2023-07-04T18:51:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.717+0000 7f8bebe61700 1 -- 172.21.15.77:0/2420650110 wait complete. 2023-07-04T18:51:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.718+0000 7f8bebe61700 1 Processor -- start 2023-07-04T18:51:59.722 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.718+0000 7f8bebe61700 1 -- start start 2023-07-04T18:51:59.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.718+0000 7f8bebe61700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.718+0000 7f8bebe61700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8be415c510 0x7f8be4147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.724 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be414b5e0 con 0x7f8be406ef20 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40ad9c0 con 0x7f8be415c510 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be40caa90 con 0x7f8be40cf960 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45234/0 (socket says 172.21.15.77:45234) 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 -- 172.21.15.77:0/1668126603 learned_addr learned my addr 172.21.15.77:0/1668126603 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be93fc700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.725 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bea3fe700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8be415c510 0x7f8be4147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 -- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 msgr2=0x7f8be40ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 -- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8be415c510 msgr2=0x7f8be4147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8be415c510 0x7f8be4147ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 -- 172.21.15.77:0/1668126603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be00094a0 con 0x7f8be406ef20 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8be9bfd700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8be000ebe0 tx=0x7f8be0009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be000c8b0 con 0x7f8be406ef20 2023-07-04T18:51:59.726 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be000ca10 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc009a70 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be000cda0 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.719+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be40ad090 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.720+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be404e210 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.721+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8be0013470 con 0x7f8be406ef20 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.721+0000 7f8bda7fc700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 0x7f8bc806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:51:59.727 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.721+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f8be0089620 con 0x7f8be406ef20 2023-07-04T18:51:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.721+0000 7f8be93fc700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 0x7f8bc806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:51:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.722+0000 7f8be93fc700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 0x7f8bc806d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8be415fce0 tx=0x7f8bd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:51:59.728 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.726+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8be00546d0 con 0x7f8be406ef20 2023-07-04T18:51:59.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.911+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8be40adb00 con 0x7f8be406ef20 2023-07-04T18:51:59.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.912+0000 7f8bda7fc700 1 -- 172.21.15.77:0/1668126603 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f8be0057cf0 con 0x7f8be406ef20 2023-07-04T18:51:59.914 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:51:59.914 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:51:59.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 msgr2=0x7f8bc806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 0x7f8bc806d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8be415fce0 tx=0x7f8bd4009040 comp rx=0 tx=0).stop 2023-07-04T18:51:59.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 msgr2=0x7f8be40ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:51:59.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8be000ebe0 tx=0x7f8be0009e20 comp rx=0 tx=0).stop 2023-07-04T18:51:59.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 shutdown_connections 2023-07-04T18:51:59.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f8bc806ace0 0x7f8bc806d190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8be406ef20 0x7f8be40ca550 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8be415c510 0x7f8be4147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 --2- 172.21.15.77:0/1668126603 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f8be40cf960 0x7f8be40ae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:51:59.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 >> 172.21.15.77:0/1668126603 conn(0x7f8be406d7c0 msgr2=0x7f8be40cc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:51:59.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 shutdown_connections 2023-07-04T18:51:59.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:51:59.915+0000 7f8bebe61700 1 -- 172.21.15.77:0/1668126603 wait complete. 2023-07-04T18:51:59.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:51:59.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:52:00.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.056+0000 7fabc6b7e700 1 Processor -- start 2023-07-04T18:52:00.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.056+0000 7fabc6b7e700 1 -- start start 2023-07-04T18:52:00.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabc6b7e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabc015d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabc6b7e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc015da80 con 0x7fabc0160a80 2023-07-04T18:52:00.061 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabc6b7e700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00ac8a0 con 0x7fabc006a550 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabc6b7e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc015d0b0 con 0x7fabc006ea90 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabbffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fabc0160a80 0x7fabc0069e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52380/0 (socket says 172.21.15.77:52380) 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabbffff700 1 -- 172.21.15.77:0/1725009644 learned_addr learned my addr 172.21.15.77:0/1725009644 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabc491a700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabc015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898416376 0 0) 0x7fabac001110 con 0x7fabc0160a80 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.057+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb00035b0 con 0x7fabc0160a80 2023-07-04T18:52:00.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2707108971 0 0) 0x7fabb00035b0 con 0x7fabc0160a80 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 msgr2=0x7fabc015d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabc015d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 >> v1:172.21.15.82:6789/0 conn(0x7fabc006a550 legacy=0x7fabc00b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc015ca80 con 0x7fabc0160a80 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fabac001800 con 0x7fabc0160a80 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 >> v1:172.21.15.77:6789/0 conn(0x7fabc0160a80 legacy=0x7fabc0069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0003a30 0x7fabb0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabb0004850 0x7fabb0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabb0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc015d0b0 con 0x7fabb0004850 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb000bb10 con 0x7fabb0003a30 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb000bcd0 con 0x7fabc006ea90 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabb0004850 0x7fabb0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabc491a700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0003a30 0x7fabb0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabc511b700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabb0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 -- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 msgr2=0x7fabb0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc006ea90 0x7fabb0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 -- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0003a30 msgr2=0x7fabb0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0003a30 0x7fabb0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 -- 172.21.15.77:0/1725009644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb00035b0 con 0x7fabb0004850 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.058+0000 7fabbffff700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabb0004850 0x7fabb0004cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabac00c210 tx=0x7fabb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6e1df0472a2a97 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac007d90 con 0x7fabb0004850 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac00cdd0 con 0x7fabb0004850 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabbf7fe700 1 -- 172.21.15.77:0/1725009644 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac005ba0 con 0x7fabb0004850 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 -- 172.21.15.77:0/1725009644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabb0004850 msgr2=0x7fabb0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/1725009644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabb0004850 0x7fabb0004cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabac00c210 tx=0x7fabb0005880 comp rx=0 tx=0).stop 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc511b700 1 -- 172.21.15.77:0/1725009644 reap_dead start 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 -- 172.21.15.77:0/1725009644 shutdown_connections 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 -- 172.21.15.77:0/1725009644 >> 172.21.15.77:0/1725009644 conn(0x7fabc006af50 msgr2=0x7fabc006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 -- 172.21.15.77:0/1725009644 shutdown_connections 2023-07-04T18:52:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.059+0000 7fabc6b7e700 1 -- 172.21.15.77:0/1725009644 wait complete. 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.060+0000 7fabc6b7e700 1 Processor -- start 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.060+0000 7fabc6b7e700 1 -- start start 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0004850 0x7fabc01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc0154630 0x7fabc00cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01e5e10 con 0x7fabc01e8150 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc0153e20 con 0x7fabb0004850 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabc6b7e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01e8e80 con 0x7fabc0154630 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabbffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabbffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45254/0 (socket says 172.21.15.77:45254) 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.061+0000 7fabbffff700 1 -- 172.21.15.77:0/2766187315 learned_addr learned my addr 172.21.15.77:0/2766187315 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabc491a700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0004850 0x7fabc01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabc511b700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc0154630 0x7fabc00cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 -- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc0154630 msgr2=0x7fabc00cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc0154630 0x7fabc00cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 -- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0004850 msgr2=0x7fabc01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0004850 0x7fabc01538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 -- 172.21.15.77:0/2766187315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00cc1a0 con 0x7fabc01e8150 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbffff700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabc015d1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac0015c0 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac001720 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc01e85e0 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac007310 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.062+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc01e8ba0 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.063+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc01e5e10 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.063+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fabac019030 con 0x7fabc01e8150 2023-07-04T18:52:00.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.064+0000 7fabbd7fa700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 0x7faba406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.064+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fabac08a780 con 0x7fabc01e8150 2023-07-04T18:52:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.064+0000 7fabc491a700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 0x7faba406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.064+0000 7fabc491a700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 0x7faba406d190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009ba0 tx=0x7fabb4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.068+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fabac0558c0 con 0x7fabc01e8150 2023-07-04T18:52:00.257 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.256+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabc004e210 con 0x7fabc01e8150 2023-07-04T18:52:00.258 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.256+0000 7fabbd7fa700 1 -- 172.21.15.77:0/2766187315 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabac017020 con 0x7fabc01e8150 2023-07-04T18:52:00.258 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:52:00.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 msgr2=0x7faba406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 0x7faba406d190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009ba0 tx=0x7fabb4008040 comp rx=0 tx=0).stop 2023-07-04T18:52:00.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 msgr2=0x7fabc00cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabc015d1f0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 shutdown_connections 2023-07-04T18:52:00.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7faba406ace0 0x7faba406d190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fabc0154630 0x7fabc00cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabb0004850 0x7fabc01538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 --2- 172.21.15.77:0/2766187315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fabc01e8150 0x7fabc00cdab0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 >> 172.21.15.77:0/2766187315 conn(0x7fabc006af50 msgr2=0x7fabc00ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 shutdown_connections 2023-07-04T18:52:00.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.258+0000 7fabc6b7e700 1 -- 172.21.15.77:0/2766187315 wait complete. 2023-07-04T18:52:00.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:52:00.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.394+0000 7ff5ac15e700 1 Processor -- start 2023-07-04T18:52:00.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.394+0000 7ff5ac15e700 1 -- start start 2023-07-04T18:52:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5a4069e40 0x7ff5a406a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff5a40d01c0 0x7ff5a40ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a40ae970 0x7ff5a415db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a40cf750 con 0x7ff5a4069e40 2023-07-04T18:52:00.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a40aedb0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5ac15e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a40af060 con 0x7ff5a40ae970 2023-07-04T18:52:00.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5a4069e40 0x7ff5a406a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff5a4069e40 0x7ff5a406a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45260/0 (socket says 172.21.15.77:45260) 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 -- 172.21.15.77:0/353730169 learned_addr learned my addr 172.21.15.77:0/353730169 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 -- 172.21.15.77:0/353730169 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a40ae970 msgr2=0x7ff5a415db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/353730169 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a40ae970 0x7ff5a415db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 -- 172.21.15.77:0/353730169 >> v2:172.21.15.82:3300/0 conn(0x7ff5a40d01c0 msgr2=0x7ff5a40ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/353730169 >> v2:172.21.15.82:3300/0 conn(0x7ff5a40d01c0 0x7ff5a40ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.395+0000 7ff5a9efa700 1 -- 172.21.15.77:0/353730169 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a40cf9c0 con 0x7ff5a4069e40 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/353730169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a4069e40 0x7ff5a406a210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4070bf0 tx=0x7ff59c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9d4339cadc8a21 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5a8ef8700 1 -- 172.21.15.77:0/353730169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00e070 con 0x7ff5a4069e40 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5a8ef8700 1 -- 172.21.15.77:0/353730169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c00b9c0 con 0x7ff5a4069e40 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5a8ef8700 1 -- 172.21.15.77:0/353730169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00bd70 con 0x7ff5a4069e40 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 -- 172.21.15.77:0/353730169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a4069e40 msgr2=0x7ff5a406a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/353730169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a4069e40 0x7ff5a406a210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4070bf0 tx=0x7ff59c00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 -- 172.21.15.77:0/353730169 shutdown_connections 2023-07-04T18:52:00.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/353730169 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a40ae970 0x7ff5a415db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/353730169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a4069e40 0x7ff5a406a210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/353730169 >> v2:172.21.15.82:3300/0 conn(0x7ff5a40d01c0 0x7ff5a40ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 -- 172.21.15.77:0/353730169 >> 172.21.15.77:0/353730169 conn(0x7ff5a406af50 msgr2=0x7ff5a406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 -- 172.21.15.77:0/353730169 shutdown_connections 2023-07-04T18:52:00.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.396+0000 7ff5ac15e700 1 -- 172.21.15.77:0/353730169 wait complete. 2023-07-04T18:52:00.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.397+0000 7ff5ac15e700 1 Processor -- start 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 -- start start 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a4069e40 0x7ff5a4140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5a40ae970 0x7ff5a41e64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a41e6030 con 0x7ff5a40d01c0 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a41e6a00 con 0x7ff5a40ae970 2023-07-04T18:52:00.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5ac15e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a41e5420 con 0x7ff5a4069e40 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5aa6fb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5aa6fb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45270/0 (socket says 172.21.15.77:45270) 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.398+0000 7ff5aa6fb700 1 -- 172.21.15.77:0/284680322 learned_addr learned my addr 172.21.15.77:0/284680322 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a4069e40 0x7ff5a4140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 -- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a4069e40 msgr2=0x7ff5a4140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a4069e40 0x7ff5a4140260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5a96f9700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5a40ae970 0x7ff5a41e64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 -- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5a40ae970 msgr2=0x7ff5a41e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5a40ae970 0x7ff5a41e64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 -- 172.21.15.77:0/284680322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c009b30 con 0x7ff5a40d01c0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5aa6fb700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000b440 tx=0x7ff5a000b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0011830 con 0x7ff5a40d01c0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a00119b0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a41e5700 con 0x7ff5a40d01c0 2023-07-04T18:52:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0011d60 con 0x7ff5a40d01c0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.399+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a41e5cc0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.400+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a41e61e0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.400+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff5a001a400 con 0x7ff5a40d01c0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.401+0000 7ff59a7fc700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 0x7ff58806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.401+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff5a0089430 con 0x7ff5a40d01c0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.401+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 0x7ff58806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.401+0000 7ff5a9efa700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 0x7ff58806d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff59c000f80 tx=0x7ff59c009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.405+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff5a41e61e0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.590+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff5a41e61e0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.590+0000 7ff59a7fc700 1 -- 172.21.15.77:0/284680322 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff5a41e61e0 con 0x7ff5a40d01c0 2023-07-04T18:52:00.592 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:52:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 msgr2=0x7ff58806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 0x7ff58806d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff59c000f80 tx=0x7ff59c009000 comp rx=0 tx=0).stop 2023-07-04T18:52:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 msgr2=0x7ff5a4143cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000b440 tx=0x7ff5a000b800 comp rx=0 tx=0).stop 2023-07-04T18:52:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 shutdown_connections 2023-07-04T18:52:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff58806ace0 0x7ff58806d190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5a40d01c0 0x7ff5a4143cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff5a4069e40 0x7ff5a4140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 --2- 172.21.15.77:0/284680322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff5a40ae970 0x7ff5a41e64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 >> 172.21.15.77:0/284680322 conn(0x7ff5a406af50 msgr2=0x7ff5a40af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 shutdown_connections 2023-07-04T18:52:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.592+0000 7ff5ac15e700 1 -- 172.21.15.77:0/284680322 wait complete. 2023-07-04T18:52:00.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:52:00.729 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.726+0000 7ff3a49ab700 1 Processor -- start 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.726+0000 7ff3a49ab700 1 -- start start 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff3a49ab700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff3a006ec20 0x7ff3a015d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff3a49ab700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff3a0160a60 0x7ff3a0069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff3a49ab700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a015da60 con 0x7ff3a006a380 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff3a49ab700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a006eff0 con 0x7ff3a0160a60 2023-07-04T18:52:00.730 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff3a49ab700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a00ac8a0 con 0x7ff3a006ec20 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.727+0000 7ff39dd9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff3a0160a60 0x7ff3a0069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff3a0160a60 0x7ff3a0069e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45278/0 (socket says 172.21.15.77:45278) 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 -- 172.21.15.77:0/1599353385 learned_addr learned my addr 172.21.15.77:0/1599353385 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39e59c700 1 --2- 172.21.15.77:0/1599353385 >> v2:172.21.15.82:3300/0 conn(0x7ff3a006ec20 0x7ff3a015d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 -- 172.21.15.77:0/1599353385 >> v2:172.21.15.82:3300/0 conn(0x7ff3a006ec20 msgr2=0x7ff3a015d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 --2- 172.21.15.77:0/1599353385 >> v2:172.21.15.82:3300/0 conn(0x7ff3a006ec20 0x7ff3a015d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.731 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 -- 172.21.15.77:0/1599353385 >> v1:172.21.15.82:6789/0 conn(0x7ff3a006a380 legacy=0x7ff3a00b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 -- 172.21.15.77:0/1599353385 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a015d090 con 0x7ff3a0160a60 2023-07-04T18:52:00.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39dd9b700 1 --2- 172.21.15.77:0/1599353385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a0160a60 0x7ff3a0069e40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00710b0 tx=0x7ff390009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8dbe7db3770e95 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39d59a700 1 -- 172.21.15.77:0/1599353385 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000b870 con 0x7ff3a0160a60 2023-07-04T18:52:00.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39d59a700 1 -- 172.21.15.77:0/1599353385 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39000b9d0 con 0x7ff3a0160a60 2023-07-04T18:52:00.732 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.728+0000 7ff39d59a700 1 -- 172.21.15.77:0/1599353385 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000bd60 con 0x7ff3a0160a60 2023-07-04T18:52:00.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 -- 172.21.15.77:0/1599353385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a0160a60 msgr2=0x7ff3a0069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/1599353385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a0160a60 0x7ff3a0069e40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00710b0 tx=0x7ff390009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.736 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 -- 172.21.15.77:0/1599353385 shutdown_connections 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/1599353385 >> v2:172.21.15.82:3300/0 conn(0x7ff3a006ec20 0x7ff3a015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/1599353385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a0160a60 0x7ff3a0069e40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 -- 172.21.15.77:0/1599353385 >> 172.21.15.77:0/1599353385 conn(0x7ff3a006af50 msgr2=0x7ff3a006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 -- 172.21.15.77:0/1599353385 shutdown_connections 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.729+0000 7ff3a49ab700 1 -- 172.21.15.77:0/1599353385 wait complete. 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.730+0000 7ff3a49ab700 1 Processor -- start 2023-07-04T18:52:00.737 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.730+0000 7ff3a49ab700 1 -- start start 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 0x7ff3a01426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff3a0160a60 0x7ff3a00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0143300 con 0x7ff3a006ec20 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0159920 con 0x7ff3a006a380 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff3a49ab700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0159c00 con 0x7ff3a0160a60 2023-07-04T18:52:00.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45280/0 (socket says 172.21.15.77:45280) 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 -- 172.21.15.77:0/425561503 learned_addr learned my addr 172.21.15.77:0/425561503 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39e59c700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 0x7ff3a01426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39ed9d700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff3a0160a60 0x7ff3a00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 -- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff3a0160a60 msgr2=0x7ff3a00b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff3a0160a60 0x7ff3a00b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 -- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 msgr2=0x7ff3a01426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 0x7ff3a01426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 -- 172.21.15.77:0/425561503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390009550 con 0x7ff3a006ec20 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39e59c700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 0x7ff3a01426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff39dd9b700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff390009c10 tx=0x7ff390004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3900036a0 con 0x7ff3a006ec20 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.731+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff390003800 con 0x7ff3a006ec20 2023-07-04T18:52:00.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.732+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a0159ee0 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.732+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390003b90 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.732+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0158c30 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.733+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0073790 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.733+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff390003cf0 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.733+0000 7ff38effd700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.734+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff39008a980 con 0x7ff3a006ec20 2023-07-04T18:52:00.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.734+0000 7ff39e59c700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:00.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.734+0000 7ff39e59c700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 0x7ff38006d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff39400ba70 tx=0x7ff394008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:00.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.738+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff390055be0 con 0x7ff3a006ec20 2023-07-04T18:52:00.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.922+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3a0143440 con 0x7ff3a006ec20 2023-07-04T18:52:00.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.922+0000 7ff38effd700 1 -- 172.21.15.77:0/425561503 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3a0143440 con 0x7ff3a006ec20 2023-07-04T18:52:00.924 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:52:00.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 msgr2=0x7ff38006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 0x7ff38006d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff39400ba70 tx=0x7ff394008040 comp rx=0 tx=0).stop 2023-07-04T18:52:00.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 msgr2=0x7ff3a00b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:00.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff390009c10 tx=0x7ff390004750 comp rx=0 tx=0).stop 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 shutdown_connections 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff38006ace0 0x7ff38006d190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff3a0160a60 0x7ff3a00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a006a380 0x7ff3a01426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 --2- 172.21.15.77:0/425561503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3a006ec20 0x7ff3a00b17c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 >> 172.21.15.77:0/425561503 conn(0x7ff3a006af50 msgr2=0x7ff3a006d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:00.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 shutdown_connections 2023-07-04T18:52:00.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:00.924+0000 7ff3a49ab700 1 -- 172.21.15.77:0/425561503 wait complete. 2023-07-04T18:52:00.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:52:01.062 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 Processor -- start 2023-07-04T18:52:01.063 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 -- start start 2023-07-04T18:52:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc15d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f19dc06a550 0x7f19dc0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc15da60 con 0x7f19dc160a60 2023-07-04T18:52:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc06eff0 con 0x7f19dc06a550 2023-07-04T18:52:01.064 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.061+0000 7f19e30b0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0ac8a0 con 0x7f19dc06ec20 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc15d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e164d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f19dc06a550 0x7f19dc0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19dbfff700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f19dc160a60 0x7f19dc069e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60336/0 (socket says 172.21.15.77:60336) 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19dbfff700 1 -- 172.21.15.77:0/300057527 learned_addr learned my addr 172.21.15.77:0/300057527 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 -- 172.21.15.77:0/300057527 >> v2:172.21.15.82:3300/0 conn(0x7f19dc06a550 msgr2=0x7f19dc0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 --2- 172.21.15.77:0/300057527 >> v2:172.21.15.82:3300/0 conn(0x7f19dc06a550 0x7f19dc0b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.065 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 -- 172.21.15.77:0/300057527 >> v1:172.21.15.82:6789/0 conn(0x7f19dc160a60 legacy=0x7f19dc069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 -- 172.21.15.77:0/300057527 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc15d090 con 0x7f19dc06ec20 2023-07-04T18:52:01.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19e0e4c700 1 --2- 172.21.15.77:0/300057527 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc15d520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009840 tx=0x7f19d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8381e731fcddbce4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.062+0000 7f19db7fe700 1 -- 172.21.15.77:0/300057527 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0011830 con 0x7f19dc06ec20 2023-07-04T18:52:01.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19db7fe700 1 -- 172.21.15.77:0/300057527 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0011990 con 0x7f19dc06ec20 2023-07-04T18:52:01.066 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19db7fe700 1 -- 172.21.15.77:0/300057527 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0011d20 con 0x7f19dc06ec20 2023-07-04T18:52:01.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 -- 172.21.15.77:0/300057527 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 msgr2=0x7f19dc15d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 --2- 172.21.15.77:0/300057527 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc15d520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009840 tx=0x7f19d0009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:01.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 -- 172.21.15.77:0/300057527 shutdown_connections 2023-07-04T18:52:01.068 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 --2- 172.21.15.77:0/300057527 >> v2:172.21.15.82:3300/0 conn(0x7f19dc06a550 0x7f19dc0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 --2- 172.21.15.77:0/300057527 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc15d520 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 -- 172.21.15.77:0/300057527 >> 172.21.15.77:0/300057527 conn(0x7f19dc06af50 msgr2=0x7f19dc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 -- 172.21.15.77:0/300057527 shutdown_connections 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.063+0000 7f19e30b0700 1 -- 172.21.15.77:0/300057527 wait complete. 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.064+0000 7f19e30b0700 1 Processor -- start 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.064+0000 7f19e30b0700 1 -- start start 2023-07-04T18:52:01.069 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19dc06a550 0x7f19dc0cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc1452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b00d0 con 0x7f19dc160a60 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b0ed0 con 0x7f19dc06a550 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e30b0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0cc840 con 0x7f19dc06ec20 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e164d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e164d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45296/0 (socket says 172.21.15.77:45296) 2023-07-04T18:52:01.070 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19e164d700 1 -- 172.21.15.77:0/3886573159 learned_addr learned my addr 172.21.15.77:0/3886573159 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.065+0000 7f19dbfff700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc1452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e0e4c700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19dc06a550 0x7f19dc0cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 -- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 msgr2=0x7f19dc1452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc1452b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 -- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19dc06a550 msgr2=0x7f19dc0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19dc06a550 0x7f19dc0cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 -- 172.21.15.77:0/3886573159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00094a0 con 0x7f19dc160a60 2023-07-04T18:52:01.071 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e164d700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f19d400efc0 tx=0x7f19d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4009b20 con 0x7f19dc160a60 2023-07-04T18:52:01.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d400f040 con 0x7f19dc160a60 2023-07-04T18:52:01.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0b02c0 con 0x7f19dc160a60 2023-07-04T18:52:01.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4009e50 con 0x7f19dc160a60 2023-07-04T18:52:01.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.066+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc0b0880 con 0x7f19dc160a60 2023-07-04T18:52:01.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.067+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0b1060 con 0x7f19dc160a60 2023-07-04T18:52:01.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.067+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f19d4014480 con 0x7f19dc160a60 2023-07-04T18:52:01.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.068+0000 7f19d97fa700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 0x7f19c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.068+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f19d4089fb0 con 0x7f19dc160a60 2023-07-04T18:52:01.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.068+0000 7f19e0e4c700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 0x7f19c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.068+0000 7f19e0e4c700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 0x7f19c006d190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19d0000c00 tx=0x7f19d0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.072+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f19d40bc310 con 0x7f19dc160a60 2023-07-04T18:52:01.259 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.258+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19dc04e210 con 0x7f19dc160a60 2023-07-04T18:52:01.260 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.258+0000 7f19d97fa700 1 -- 172.21.15.77:0/3886573159 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19d40180a0 con 0x7f19dc160a60 2023-07-04T18:52:01.260 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:52:01.262 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 msgr2=0x7f19c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 0x7f19c006d190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19d0000c00 tx=0x7f19d0008030 comp rx=0 tx=0).stop 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 msgr2=0x7f19dc155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f19d400efc0 tx=0x7f19d400c3a0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 shutdown_connections 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f19c006ace0 0x7f19c006d190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19dc160a60 0x7f19dc155240 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.263 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19dc06a550 0x7f19dc0cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 --2- 172.21.15.77:0/3886573159 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f19dc06ec20 0x7f19dc1452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 >> 172.21.15.77:0/3886573159 conn(0x7f19dc06af50 msgr2=0x7f19dc069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:01.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 shutdown_connections 2023-07-04T18:52:01.264 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.260+0000 7f19e30b0700 1 -- 172.21.15.77:0/3886573159 wait complete. 2023-07-04T18:52:01.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-04T18:52:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 Processor -- start 2023-07-04T18:52:01.396 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 -- start start 2023-07-04T18:52:01.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff320070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3200cf830 con 0x7ff320070db0 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3200717f0 con 0x7ff32015c640 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.395+0000 7ff324fec700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff320071250 con 0x7ff32015ca10 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff320070db0 0x7ff3200cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52442/0 (socket says 172.21.15.77:52442) 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31ed9d700 1 -- 172.21.15.77:0/608518648 learned_addr learned my addr 172.21.15.77:0/608518648 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff320070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111551394 0 0) 0x7ff3200cf830 con 0x7ff320070db0 2023-07-04T18:52:01.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3080035b0 con 0x7ff320070db0 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860626871 0 0) 0x7ff3200717f0 con 0x7ff32015c640 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3200cf830 con 0x7ff32015c640 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729789680 0 0) 0x7ff3080035b0 con 0x7ff320070db0 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 msgr2=0x7ff320070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff320070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 >> v1:172.21.15.82:6789/0 conn(0x7ff32015c640 legacy=0x7ff3200cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3200cd290 con 0x7ff320070db0 2023-07-04T18:52:01.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7ff318001970 con 0x7ff320070db0 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff320070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 >> v1:172.21.15.77:6789/0 conn(0x7ff320070db0 legacy=0x7ff3200cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff308003bc0 0x7ff308003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff308004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff3080052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30800bba0 con 0x7ff308004850 2023-07-04T18:52:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30800bd10 con 0x7ff308003bc0 2023-07-04T18:52:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30800bed0 con 0x7ff32015ca10 2023-07-04T18:52:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff308004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31e59c700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff308003bc0 0x7ff308003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.396+0000 7ff31ed9d700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff3080052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 -- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 msgr2=0x7ff3080052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015ca10 0x7ff3080052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 -- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff308003bc0 msgr2=0x7ff308003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:01.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff308003bc0 0x7ff308003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 -- 172.21.15.77:0/608518648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100094a0 con 0x7ff308004850 2023-07-04T18:52:01.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff308004cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff310009d00 tx=0x7ff308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354e56c4ff0c9dec server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310009df0 con 0x7ff308004850 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31000e9a0 con 0x7ff308004850 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31d59a700 1 -- 172.21.15.77:0/608518648 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31000ed50 con 0x7ff308004850 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff324fec700 1 -- 172.21.15.77:0/608518648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 msgr2=0x7ff308004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff324fec700 1 --2- 172.21.15.77:0/608518648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff308004cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff310009d00 tx=0x7ff308005880 comp rx=0 tx=0).stop 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff31ed9d700 1 -- 172.21.15.77:0/608518648 reap_dead start 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff324fec700 1 -- 172.21.15.77:0/608518648 shutdown_connections 2023-07-04T18:52:01.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.397+0000 7ff324fec700 1 -- 172.21.15.77:0/608518648 >> 172.21.15.77:0/608518648 conn(0x7ff32006bba0 msgr2=0x7ff3200b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:01.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.398+0000 7ff324fec700 1 -- 172.21.15.77:0/608518648 shutdown_connections 2023-07-04T18:52:01.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.398+0000 7ff324fec700 1 -- 172.21.15.77:0/608518648 wait complete. 2023-07-04T18:52:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.399+0000 7ff324fec700 1 Processor -- start 2023-07-04T18:52:01.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.399+0000 7ff324fec700 1 -- start start 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015b910 0x7ff3200b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff32015b230 0x7ff32015efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3201f6320 con 0x7ff308004850 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3200b3990 con 0x7ff32015b230 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32015b6d0 con 0x7ff32015b910 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45312/0 (socket says 172.21.15.77:45312) 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 -- 172.21.15.77:0/1133168690 learned_addr learned my addr 172.21.15.77:0/1133168690 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015b910 0x7ff3200b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 -- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015b910 msgr2=0x7ff3200b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015b910 0x7ff3200b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 -- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff32015b230 msgr2=0x7ff32015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff32015b230 0x7ff32015efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 -- 172.21.15.77:0/1133168690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3200b3c30 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff31e59c700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff31400f770 tx=0x7ff31400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3140052e0 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314005460 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3200b3f10 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3140038c0 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.400+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3200b44d0 con 0x7ff308004850 2023-07-04T18:52:01.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.401+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32004e210 con 0x7ff308004850 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.402+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff314003a20 con 0x7ff308004850 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.402+0000 7ff30effd700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 0x7ff30006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.402+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff3140891e0 con 0x7ff308004850 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.402+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 0x7ff30006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.403+0000 7ff31dd9b700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 0x7ff30006d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff3100090b0 tx=0x7ff310008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:01.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.406+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff314054320 con 0x7ff308004850 2023-07-04T18:52:01.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.614+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff3200b2220 con 0x7ff308004850 2023-07-04T18:52:01.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.614+0000 7ff30effd700 1 -- 172.21.15.77:0/1133168690 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff314057940 con 0x7ff308004850 2023-07-04T18:52:01.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 msgr2=0x7ff30006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 0x7ff30006d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff3100090b0 tx=0x7ff310008030 comp rx=0 tx=0).stop 2023-07-04T18:52:01.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 msgr2=0x7ff3201f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:01.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff31400f770 tx=0x7ff31400fb30 comp rx=0 tx=0).stop 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 shutdown_connections 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff30006ace0 0x7ff30006d190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff308004850 0x7ff3201f5d40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff32015b230 0x7ff32015efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 --2- 172.21.15.77:0/1133168690 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff32015b910 0x7ff3200b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:01.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 >> 172.21.15.77:0/1133168690 conn(0x7ff32006bba0 msgr2=0x7ff3200acb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:01.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 shutdown_connections 2023-07-04T18:52:01.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.616+0000 7ff324fec700 1 -- 172.21.15.77:0/1133168690 wait complete. 2023-07-04T18:52:01.621 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2023-07-04T18:52:01.631 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-07-04T18:52:01.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:01.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-07-04T18:52:01.679 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T18:52:01.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:01.681 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-07-04T18:52:01.736 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:52:01.737 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:52:01.759 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:52:01.759 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:52:01.818 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T18:52:01.819 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:52:01.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:02.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.998+0000 7f58af7b9700 1 Processor -- start 2023-07-04T18:52:02.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:01.999+0000 7f58af7b9700 1 -- start start 2023-07-04T18:52:02.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58af7b9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a815ea40 0x7f58a80cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58af7b9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58a80cf920 0x7f58a806ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58af7b9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80cf100 con 0x7f58a815a430 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58af7b9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a815df60 con 0x7f58a80cf920 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58af7b9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a815e240 con 0x7f58a815ea40 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f58a80cf920 0x7f58a806ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58ad555700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f58a815a430 0x7f58a80ce6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52458/0 (socket says 172.21.15.77:52458) 2023-07-04T18:52:02.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58ad555700 1 -- 172.21.15.77:0/3822037386 learned_addr learned my addr 172.21.15.77:0/3822037386 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58acd54700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a815ea40 0x7f58a80cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 -- 172.21.15.77:0/3822037386 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a815ea40 msgr2=0x7f58a80cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a815ea40 0x7f58a80cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 -- 172.21.15.77:0/3822037386 >> v1:172.21.15.77:6789/0 conn(0x7f58a815a430 legacy=0x7f58a80ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 -- 172.21.15.77:0/3822037386 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a815e520 con 0x7f58a80cf920 2023-07-04T18:52:02.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f58add56700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80cf920 0x7f58a806ef20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009d90 tx=0x7f58a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69d4c0947639fc1 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f589ffff700 1 -- 172.21.15.77:0/3822037386 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a000e070 con 0x7f58a80cf920 2023-07-04T18:52:02.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f589ffff700 1 -- 172.21.15.77:0/3822037386 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a000bbd0 con 0x7f58a80cf920 2023-07-04T18:52:02.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.000+0000 7f589ffff700 1 -- 172.21.15.77:0/3822037386 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0012430 con 0x7f58a80cf920 2023-07-04T18:52:02.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 -- 172.21.15.77:0/3822037386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80cf920 msgr2=0x7f58a806ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80cf920 0x7f58a806ef20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009d90 tx=0x7f58a000b330 comp rx=0 tx=0).stop 2023-07-04T18:52:02.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 -- 172.21.15.77:0/3822037386 shutdown_connections 2023-07-04T18:52:02.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80cf920 0x7f58a806ef20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 --2- 172.21.15.77:0/3822037386 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a815ea40 0x7f58a80cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 -- 172.21.15.77:0/3822037386 >> 172.21.15.77:0/3822037386 conn(0x7f58a806d7c0 msgr2=0x7f58a806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 -- 172.21.15.77:0/3822037386 shutdown_connections 2023-07-04T18:52:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.001+0000 7f58af7b9700 1 -- 172.21.15.77:0/3822037386 wait complete. 2023-07-04T18:52:02.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.002+0000 7f58af7b9700 1 Processor -- start 2023-07-04T18:52:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.002+0000 7f58af7b9700 1 -- start start 2023-07-04T18:52:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a80cf920 0x7f58a8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 0x7f58a80c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81f4a10 con 0x7f58a80c5120 2023-07-04T18:52:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a80c55c0 con 0x7f58a815ea40 2023-07-04T18:52:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58af7b9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81f3d90 con 0x7f58a80cf920 2023-07-04T18:52:02.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45330/0 (socket says 172.21.15.77:45330) 2023-07-04T18:52:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 -- 172.21.15.77:0/4153103152 learned_addr learned my addr 172.21.15.77:0/4153103152 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58ad555700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a80cf920 0x7f58a8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58acd54700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 0x7f58a80c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 -- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a80cf920 msgr2=0x7f58a8154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a80cf920 0x7f58a8154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 -- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 msgr2=0x7f58a80c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 0x7f58a80c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58add56700 1 -- 172.21.15.77:0/4153103152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0009a70 con 0x7f58a80c5120 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.003+0000 7f58acd54700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 0x7f58a80c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f58add56700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f58a8071420 tx=0x7f58a0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0003ea0 con 0x7f58a80c5120 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a0020070 con 0x7f58a80c5120 2023-07-04T18:52:02.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a81f4070 con 0x7f58a80c5120 2023-07-04T18:52:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0004280 con 0x7f58a80c5120 2023-07-04T18:52:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.004+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a81f4630 con 0x7f58a80c5120 2023-07-04T18:52:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.005+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a81f4b50 con 0x7f58a80c5120 2023-07-04T18:52:02.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.005+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f58a00043e0 con 0x7f58a80c5120 2023-07-04T18:52:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.006+0000 7f589dffb700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 0x7f588c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.006+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f58a0022070 con 0x7f58a80c5120 2023-07-04T18:52:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.006+0000 7f58ad555700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 0x7f588c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.006+0000 7f58ad555700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 0x7f588c06d190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5898005910 tx=0x7f5898009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.010+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f58a81f4b50 con 0x7f58a80c5120 2023-07-04T18:52:02.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.197+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a81f4b50 con 0x7f58a80c5120 2023-07-04T18:52:02.200 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.198+0000 7f589dffb700 1 -- 172.21.15.77:0/4153103152 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+18243 (secure 0 0 0) 0x7f58a81f4b50 con 0x7f58a80c5120 2023-07-04T18:52:02.201 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:02.201 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:51:23.299649+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:02.202 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 msgr2=0x7f588c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 0x7f588c06d190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5898005910 tx=0x7f5898009040 comp rx=0 tx=0).stop 2023-07-04T18:52:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 msgr2=0x7f58a80c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f58a8071420 tx=0x7f58a0004ad0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 shutdown_connections 2023-07-04T18:52:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f588c06ace0 0x7f588c06d190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58a80c5120 0x7f58a80c6b60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f58a80cf920 0x7f58a8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.204 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 --2- 172.21.15.77:0/4153103152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58a815ea40 0x7f58a80c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 >> 172.21.15.77:0/4153103152 conn(0x7f58a806d7c0 msgr2=0x7f58a81561f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 shutdown_connections 2023-07-04T18:52:02.205 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.200+0000 7f58af7b9700 1 -- 172.21.15.77:0/4153103152 wait complete. 2023-07-04T18:52:02.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:52:02.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:52:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.341+0000 7f036bb0a700 1 Processor -- start 2023-07-04T18:52:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 -- start start 2023-07-04T18:52:02.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03641590b0 0x7f036415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f03640b3330 0x7f03640b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03640b3f60 con 0x7f036406cfc0 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036406caa0 con 0x7f03641590b0 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.342+0000 7f036bb0a700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036415dc10 con 0x7f03640b3330 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f03641590b0 0x7f036415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f036a0a7700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f036406cfc0 0x7f036406d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52474/0 (socket says 172.21.15.77:52474) 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f036a0a7700 1 -- 172.21.15.77:0/3738891889 learned_addr learned my addr 172.21.15.77:0/3738891889 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03688a4700 1 -- 172.21.15.77:0/3738891889 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937997746 0 0) 0x7f03640b3f60 con 0x7f036406cfc0 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03690a5700 1 --2- 172.21.15.77:0/3738891889 >> v2:172.21.15.82:3300/0 conn(0x7f03640b3330 0x7f03640b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03688a4700 1 -- 172.21.15.77:0/3738891889 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03500035b0 con 0x7f036406cfc0 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 -- 172.21.15.77:0/3738891889 >> v2:172.21.15.82:3300/0 conn(0x7f03640b3330 msgr2=0x7f03640b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 --2- 172.21.15.77:0/3738891889 >> v2:172.21.15.82:3300/0 conn(0x7f03640b3330 0x7f03640b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 -- 172.21.15.77:0/3738891889 >> v1:172.21.15.77:6789/0 conn(0x7f036406cfc0 legacy=0x7f036406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 -- 172.21.15.77:0/3738891889 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036406c4e0 con 0x7f03641590b0 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03698a6700 1 --2- 172.21.15.77:0/3738891889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03641590b0 0x7f036415d6d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f036406ce30 tx=0x7f0360009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6c5aa4e7b7015b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03688a4700 1 -- 172.21.15.77:0/3738891889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360011830 con 0x7f03641590b0 2023-07-04T18:52:02.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03688a4700 1 -- 172.21.15.77:0/3738891889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360011990 con 0x7f03641590b0 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.343+0000 7f03688a4700 1 -- 172.21.15.77:0/3738891889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360011d00 con 0x7f03641590b0 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 -- 172.21.15.77:0/3738891889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03641590b0 msgr2=0x7f036415d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 --2- 172.21.15.77:0/3738891889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03641590b0 0x7f036415d6d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f036406ce30 tx=0x7f0360009b20 comp rx=0 tx=0).stop 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 -- 172.21.15.77:0/3738891889 shutdown_connections 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 --2- 172.21.15.77:0/3738891889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03641590b0 0x7f036415d6d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 --2- 172.21.15.77:0/3738891889 >> v2:172.21.15.82:3300/0 conn(0x7f03640b3330 0x7f03640b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 -- 172.21.15.77:0/3738891889 >> 172.21.15.77:0/3738891889 conn(0x7f036406af50 msgr2=0x7f036406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 -- 172.21.15.77:0/3738891889 shutdown_connections 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.344+0000 7f036bb0a700 1 -- 172.21.15.77:0/3738891889 wait complete. 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.345+0000 7f036bb0a700 1 Processor -- start 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.345+0000 7f036bb0a700 1 -- start start 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 0x7f03640b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 0x7f036414aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036414bb30 con 0x7f03640b3330 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036414ff60 con 0x7f03641590b0 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0364150240 con 0x7f036406cfc0 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45340/0 (socket says 172.21.15.77:45340) 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 -- 172.21.15.77:0/897261879 learned_addr learned my addr 172.21.15.77:0/897261879 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036a0a7700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 0x7f036414aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03698a6700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 0x7f03640b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 -- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 msgr2=0x7f03640b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 0x7f03640b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 -- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 msgr2=0x7f036414aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 0x7f036414aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 -- 172.21.15.77:0/897261879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03600094a0 con 0x7f03640b3330 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036a0a7700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 0x7f036414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03698a6700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 0x7f03640b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:02.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f03690a5700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f035400c7f0 tx=0x7f035400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354011800 con 0x7f03640b3330 2023-07-04T18:52:02.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0354011960 con 0x7f03640b3330 2023-07-04T18:52:02.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0364150520 con 0x7f03640b3330 2023-07-04T18:52:02.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0354011cf0 con 0x7f03640b3330 2023-07-04T18:52:02.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.346+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0364150ae0 con 0x7f03640b3330 2023-07-04T18:52:02.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.347+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036404e210 con 0x7f03640b3330 2023-07-04T18:52:02.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.348+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0354018360 con 0x7f03640b3330 2023-07-04T18:52:02.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.348+0000 7f035a7fc700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 0x7f034806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.348+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f0354089540 con 0x7f03640b3330 2023-07-04T18:52:02.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.348+0000 7f03698a6700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 0x7f034806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.349+0000 7f03698a6700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 0x7f034806d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0360009b60 tx=0x7f036001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.352+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0354058000 con 0x7f03640b3330 2023-07-04T18:52:02.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.541+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f036406e0b0 con 0x7f03640b3330 2023-07-04T18:52:02.543 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.542+0000 7f035a7fc700 1 -- 172.21.15.77:0/897261879 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0354057b90 con 0x7f03640b3330 2023-07-04T18:52:02.544 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:52:02.545 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 msgr2=0x7f034806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 0x7f034806d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0360009b60 tx=0x7f036001c040 comp rx=0 tx=0).stop 2023-07-04T18:52:02.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 msgr2=0x7f0364151000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f035400c7f0 tx=0x7f035400cbb0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.546 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 shutdown_connections 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f034806ace0 0x7f034806d190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03641590b0 0x7f036414aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f036406cfc0 0x7f03640b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 --2- 172.21.15.77:0/897261879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03640b3330 0x7f0364151000 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 >> 172.21.15.77:0/897261879 conn(0x7f036406af50 msgr2=0x7f036415ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 shutdown_connections 2023-07-04T18:52:02.547 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.544+0000 7f036bb0a700 1 -- 172.21.15.77:0/897261879 wait complete. 2023-07-04T18:52:02.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:52:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.679+0000 7fa5df012700 1 Processor -- start 2023-07-04T18:52:02.681 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.679+0000 7fa5df012700 1 -- start start 2023-07-04T18:52:02.683 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5df012700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d8153600 0x7fa5d8070530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5df012700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5d80d0420 0x7fa5d8069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5df012700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80cff90 con 0x7fa5d806a380 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5df012700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d81539d0 con 0x7fa5d80d0420 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5df012700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d806a7c0 con 0x7fa5d8153600 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5d7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa5d80d0420 0x7fa5d8069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5dd5af700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa5d806a380 0x7fa5d8160070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52488/0 (socket says 172.21.15.77:52488) 2023-07-04T18:52:02.684 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5dd5af700 1 -- 172.21.15.77:0/1736657016 learned_addr learned my addr 172.21.15.77:0/1736657016 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.680+0000 7fa5dcdae700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d8153600 0x7fa5d8070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d7fff700 1 -- 172.21.15.77:0/1736657016 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d8153600 msgr2=0x7fa5d8070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d7fff700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d8153600 0x7fa5d8070530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d7fff700 1 -- 172.21.15.77:0/1736657016 >> v1:172.21.15.77:6789/0 conn(0x7fa5d806a380 legacy=0x7fa5d8160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d7fff700 1 -- 172.21.15.77:0/1736657016 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80cce50 con 0x7fa5d80d0420 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d77fe700 1 -- 172.21.15.77:0/1736657016 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227431487 0 0) 0x7fa5d80cff90 con 0x7fa5d806a380 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d7fff700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d80d0420 0x7fa5d8069e40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009840 tx=0x7fa5c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647f942255afe728 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.685 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d77fe700 1 -- 172.21.15.77:0/1736657016 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4011830 con 0x7fa5d80d0420 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d77fe700 1 -- 172.21.15.77:0/1736657016 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c4011990 con 0x7fa5d80d0420 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5d77fe700 1 -- 172.21.15.77:0/1736657016 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4011d20 con 0x7fa5d80d0420 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5df012700 1 -- 172.21.15.77:0/1736657016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d80d0420 msgr2=0x7fa5d8069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.681+0000 7fa5df012700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d80d0420 0x7fa5d8069e40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009840 tx=0x7fa5c4009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 -- 172.21.15.77:0/1736657016 shutdown_connections 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d8153600 0x7fa5d8070530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 --2- 172.21.15.77:0/1736657016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d80d0420 0x7fa5d8069e40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 -- 172.21.15.77:0/1736657016 >> 172.21.15.77:0/1736657016 conn(0x7fa5d806af50 msgr2=0x7fa5d806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 -- 172.21.15.77:0/1736657016 shutdown_connections 2023-07-04T18:52:02.687 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.682+0000 7fa5df012700 1 -- 172.21.15.77:0/1736657016 wait complete. 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.683+0000 7fa5df012700 1 Processor -- start 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.683+0000 7fa5df012700 1 -- start start 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d80d0420 0x7fa5d80cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5d8153600 0x7fa5d815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.689 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81590a0 con 0x7fa5d806a380 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81592d0 con 0x7fa5d8153600 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81595b0 con 0x7fa5d80d0420 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45352/0 (socket says 172.21.15.77:45352) 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 -- 172.21.15.77:0/1681190102 learned_addr learned my addr 172.21.15.77:0/1681190102 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:02.690 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 -- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d80d0420 msgr2=0x7fa5d80cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d80d0420 0x7fa5d80cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 -- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5d8153600 msgr2=0x7fa5d815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5d8153600 0x7fa5d815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 -- 172.21.15.77:0/1681190102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40094a0 con 0x7fa5d806a380 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5dcdae700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00e870 tx=0x7fa5cc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc009590 con 0x7fa5d806a380 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc00ee70 con 0x7fa5d806a380 2023-07-04T18:52:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00ca50 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d8159890 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.684+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d8159e50 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.685+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d804e210 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.686+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa5cc00cbb0 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.686+0000 7fa5d57fa700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 0x7fa5bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.686+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fa5cc0893c0 con 0x7fa5d806a380 2023-07-04T18:52:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.686+0000 7fa5d7fff700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 0x7fa5bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.687+0000 7fa5d7fff700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 0x7fa5bc06d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80cd5e0 tx=0x7fa5c4010000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.690+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5cc057f90 con 0x7fa5d806a380 2023-07-04T18:52:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.874+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa5d80cc850 con 0x7fa5d806a380 2023-07-04T18:52:02.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.874+0000 7fa5d57fa700 1 -- 172.21.15.77:0/1681190102 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa5cc00ce60 con 0x7fa5d806a380 2023-07-04T18:52:02.876 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:52:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 msgr2=0x7fa5bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 0x7fa5bc06d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80cd5e0 tx=0x7fa5c4010000 comp rx=0 tx=0).stop 2023-07-04T18:52:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 msgr2=0x7fa5d814a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00e870 tx=0x7fa5cc00ec30 comp rx=0 tx=0).stop 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 shutdown_connections 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa5bc06ace0 0x7fa5bc06d190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5d806a380 0x7fa5d814a220 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5d8153600 0x7fa5d815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 --2- 172.21.15.77:0/1681190102 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa5d80d0420 0x7fa5d80cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 >> 172.21.15.77:0/1681190102 conn(0x7fa5d806af50 msgr2=0x7fa5d806b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 shutdown_connections 2023-07-04T18:52:02.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:02.876+0000 7fa5df012700 1 -- 172.21.15.77:0/1681190102 wait complete. 2023-07-04T18:52:02.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:52:03.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.011+0000 7efefe7e9700 1 Processor -- start 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.011+0000 7efefe7e9700 1 -- start start 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefe7e9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efef81590b0 0x7efef815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefe7e9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80b3f60 con 0x7efef806cfc0 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefe7e9700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80b4b00 con 0x7efef80b3330 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefe7e9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef806caa0 con 0x7efef81590b0 2023-07-04T18:52:03.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efef7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efef81590b0 0x7efef815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefcd86700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efef806cfc0 0x7efef806d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52500/0 (socket says 172.21.15.77:52500) 2023-07-04T18:52:03.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efefcd86700 1 -- 172.21.15.77:0/1490440260 learned_addr learned my addr 172.21.15.77:0/1490440260 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:03.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efef7fff700 1 -- 172.21.15.77:0/1490440260 >> v1:172.21.15.82:6789/0 conn(0x7efef80b3330 legacy=0x7efef80b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efef7fff700 1 -- 172.21.15.77:0/1490440260 >> v1:172.21.15.77:6789/0 conn(0x7efef806cfc0 legacy=0x7efef806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efef7fff700 1 -- 172.21.15.77:0/1490440260 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef815dc10 con 0x7efef81590b0 2023-07-04T18:52:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.012+0000 7efef6ffd700 1 -- 172.21.15.77:0/1490440260 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963985904 0 0) 0x7efef80b3f60 con 0x7efef806cfc0 2023-07-04T18:52:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efef7fff700 1 --2- 172.21.15.77:0/1490440260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef815d6d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efeec009840 tx=0x7efeec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa7a6f6dee9022b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efef6ffd700 1 -- 172.21.15.77:0/1490440260 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec004230 con 0x7efef81590b0 2023-07-04T18:52:03.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efef6ffd700 1 -- 172.21.15.77:0/1490440260 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec004390 con 0x7efef81590b0 2023-07-04T18:52:03.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efef6ffd700 1 -- 172.21.15.77:0/1490440260 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0036a0 con 0x7efef81590b0 2023-07-04T18:52:03.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 -- 172.21.15.77:0/1490440260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 msgr2=0x7efef815d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 --2- 172.21.15.77:0/1490440260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef815d6d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efeec009840 tx=0x7efeec009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 -- 172.21.15.77:0/1490440260 shutdown_connections 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 --2- 172.21.15.77:0/1490440260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef815d6d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 -- 172.21.15.77:0/1490440260 >> 172.21.15.77:0/1490440260 conn(0x7efef806af50 msgr2=0x7efef806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 -- 172.21.15.77:0/1490440260 shutdown_connections 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.013+0000 7efefe7e9700 1 -- 172.21.15.77:0/1490440260 wait complete. 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.014+0000 7efefe7e9700 1 Processor -- start 2023-07-04T18:52:03.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 -- start start 2023-07-04T18:52:03.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efef806cfc0 0x7efef815b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 0x7efef81e3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81409c0 con 0x7efef81590b0 2023-07-04T18:52:03.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8142ed0 con 0x7efef80b3330 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.015+0000 7efefe7e9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef8140020 con 0x7efef806cfc0 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45368/0 (socket says 172.21.15.77:45368) 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 -- 172.21.15.77:0/673656768 learned_addr learned my addr 172.21.15.77:0/673656768 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef7fff700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efef806cfc0 0x7efef815b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef77fe700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 0x7efef81e3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 -- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efef806cfc0 msgr2=0x7efef815b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.025 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efef806cfc0 0x7efef815b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 -- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 msgr2=0x7efef81e3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 0x7efef81e3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 -- 172.21.15.77:0/673656768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0094a0 con 0x7efef81590b0 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef77fe700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 0x7efef81e3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefcd86700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efef000cca0 tx=0x7efef000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef000ac20 con 0x7efef81590b0 2023-07-04T18:52:03.026 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef000ad80 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0007310 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef8140300 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.016+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef8159480 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.017+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef8140b00 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.018+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efef0007470 con 0x7efef81590b0 2023-07-04T18:52:03.027 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.018+0000 7efef4ff9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 0x7efee406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.018+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efef0089030 con 0x7efef81590b0 2023-07-04T18:52:03.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.018+0000 7efef7fff700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 0x7efee406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.019+0000 7efef7fff700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 0x7efee406d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efeec0090b0 tx=0x7efeec008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.028 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.022+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efef8140b00 con 0x7efef81590b0 2023-07-04T18:52:03.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.209+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efef8140b00 con 0x7efef81590b0 2023-07-04T18:52:03.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.210+0000 7efef4ff9700 1 -- 172.21.15.77:0/673656768 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7efef8140b00 con 0x7efef81590b0 2023-07-04T18:52:03.212 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:52:03.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.211+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 msgr2=0x7efee406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.211+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 0x7efee406d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efeec0090b0 tx=0x7efeec008030 comp rx=0 tx=0).stop 2023-07-04T18:52:03.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 msgr2=0x7efef81e4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efef000cca0 tx=0x7efef000a060 comp rx=0 tx=0).stop 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 shutdown_connections 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efee406ace0 0x7efee406d190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efef81590b0 0x7efef81e4230 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efef806cfc0 0x7efef815b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.214 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 --2- 172.21.15.77:0/673656768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efef80b3330 0x7efef81e3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 >> 172.21.15.77:0/673656768 conn(0x7efef806af50 msgr2=0x7efef80b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 shutdown_connections 2023-07-04T18:52:03.215 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.212+0000 7efefe7e9700 1 -- 172.21.15.77:0/673656768 wait complete. 2023-07-04T18:52:03.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:52:03.348 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.347+0000 7f6402279700 1 Processor -- start 2023-07-04T18:52:03.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.347+0000 7f6402279700 1 -- start start 2023-07-04T18:52:03.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6402279700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 0x7f63fc0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6402279700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc0ac8a0 con 0x7f63fc06ea90 2023-07-04T18:52:03.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6402279700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc15d0b0 con 0x7f63fc15d540 2023-07-04T18:52:03.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6402279700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc15ca80 con 0x7f63fc06a550 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63fbfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f63fc06ea90 0x7f63fc160a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52510/0 (socket says 172.21.15.77:52510) 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63fbfff700 1 -- 172.21.15.77:0/3993147889 learned_addr learned my addr 172.21.15.77:0/3993147889 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6400816700 1 --2- 172.21.15.77:0/3993147889 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 0x7f63fc0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894088673 0 0) 0x7f63fc0ac8a0 con 0x7f63fc06ea90 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e40035b0 con 0x7f63fc06ea90 2023-07-04T18:52:03.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161232378 0 0) 0x7f63fc15d0b0 con 0x7f63fc15d540 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc0ac8a0 con 0x7f63fc15d540 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993394565 0 0) 0x7f63e40035b0 con 0x7f63fc06ea90 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6400816700 1 -- 172.21.15.77:0/3993147889 >> v1:172.21.15.82:6789/0 conn(0x7f63fc15d540 legacy=0x7f63fc069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6400816700 1 -- 172.21.15.77:0/3993147889 >> v1:172.21.15.77:6789/0 conn(0x7f63fc06ea90 legacy=0x7f63fc160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.348+0000 7f6400816700 1 -- 172.21.15.77:0/3993147889 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0acaf0 con 0x7f63fc06a550 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6400816700 1 --2- 172.21.15.77:0/3993147889 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 0x7f63fc0b2db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009dc0 tx=0x7f63f400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499509f809301bc3 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400e070 con 0x7f63fc06a550 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003a20 con 0x7f63fc06a550 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f63faffd700 1 -- 172.21.15.77:0/3993147889 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0001670 con 0x7f63fc06a550 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 -- 172.21.15.77:0/3993147889 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 msgr2=0x7f63fc0b2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 --2- 172.21.15.77:0/3993147889 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 0x7f63fc0b2db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009dc0 tx=0x7f63f400b330 comp rx=0 tx=0).stop 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 -- 172.21.15.77:0/3993147889 shutdown_connections 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 --2- 172.21.15.77:0/3993147889 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc06a550 0x7f63fc0b2db0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 -- 172.21.15.77:0/3993147889 >> 172.21.15.77:0/3993147889 conn(0x7f63fc06af50 msgr2=0x7f63fc06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 -- 172.21.15.77:0/3993147889 shutdown_connections 2023-07-04T18:52:03.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.349+0000 7f6402279700 1 -- 172.21.15.77:0/3993147889 wait complete. 2023-07-04T18:52:03.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.350+0000 7f6402279700 1 Processor -- start 2023-07-04T18:52:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 -- start start 2023-07-04T18:52:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 0x7f63fc1464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc15d540 0x7f63fc0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc06caa0 con 0x7f63fc06ea90 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc146a30 con 0x7f63fc06a550 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.351+0000 7f6402279700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0cfd90 con 0x7f63fc15d540 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45380/0 (socket says 172.21.15.77:45380) 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 -- 172.21.15.77:0/4141643917 learned_addr learned my addr 172.21.15.77:0/4141643917 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fbfff700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 0x7f63fc1464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f6400816700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc15d540 0x7f63fc0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 -- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc15d540 msgr2=0x7f63fc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc15d540 0x7f63fc0b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 -- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 msgr2=0x7f63fc1464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 0x7f63fc1464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 -- 172.21.15.77:0/4141643917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4009a70 con 0x7f63fc06ea90 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fbfff700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 0x7f63fc1464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:03.362 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63fb7fe700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00c480 tx=0x7f63ec00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec0052d0 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec005430 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc0d0070 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec0038c0 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.352+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc06c730 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.353+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc06cc10 con 0x7f63fc06ea90 2023-07-04T18:52:03.363 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.354+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f63ec003a20 con 0x7f63fc06ea90 2023-07-04T18:52:03.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.354+0000 7f63f8ff9700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 0x7f63e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.354+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f63ec0889d0 con 0x7f63fc06ea90 2023-07-04T18:52:03.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.354+0000 7f63fbfff700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 0x7f63e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.355+0000 7f63fbfff700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 0x7f63e806d190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63f00079b0 tx=0x7f63f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.364 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.358+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f63ec053a00 con 0x7f63fc06ea90 2023-07-04T18:52:03.549 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.547+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63fc04e210 con 0x7f63fc06ea90 2023-07-04T18:52:03.550 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.548+0000 7f63f8ff9700 1 -- 172.21.15.77:0/4141643917 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63ec057020 con 0x7f63fc06ea90 2023-07-04T18:52:03.550 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:52:03.551 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 msgr2=0x7f63e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 0x7f63e806d190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63f00079b0 tx=0x7f63f0009040 comp rx=0 tx=0).stop 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 msgr2=0x7f63fc0cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00c480 tx=0x7f63ec00c840 comp rx=0 tx=0).stop 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 shutdown_connections 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f63e806ace0 0x7f63e806d190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f63fc15d540 0x7f63fc0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc06a550 0x7f63fc1464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 --2- 172.21.15.77:0/4141643917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc06ea90 0x7f63fc0cf420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 >> 172.21.15.77:0/4141643917 conn(0x7f63fc06af50 msgr2=0x7f63fc069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 shutdown_connections 2023-07-04T18:52:03.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.550+0000 7f6402279700 1 -- 172.21.15.77:0/4141643917 wait complete. 2023-07-04T18:52:03.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-04T18:52:03.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.686+0000 7efd9c0e6700 1 Processor -- start 2023-07-04T18:52:03.688 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9c0e6700 1 -- start start 2023-07-04T18:52:03.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9c0e6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd940cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9c0e6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940717f0 con 0x7efd94070f80 2023-07-04T18:52:03.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9c0e6700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940cd290 con 0x7efd9415ca10 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9c0e6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ccfb0 con 0x7efd9415c640 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9a683700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efd94070f80 0x7efd940cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52524/0 (socket says 172.21.15.77:52524) 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.687+0000 7efd9a683700 1 -- 172.21.15.77:0/1018485167 learned_addr learned my addr 172.21.15.77:0/1018485167 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99e82700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd940cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899091625 0 0) 0x7efd940717f0 con 0x7efd94070f80 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800035b0 con 0x7efd94070f80 2023-07-04T18:52:03.693 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717526072 0 0) 0x7efd800035b0 con 0x7efd94070f80 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 msgr2=0x7efd940cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd940cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 >> v1:172.21.15.82:6789/0 conn(0x7efd9415ca10 legacy=0x7efd94070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940ce640 con 0x7efd94070f80 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7efd8c0019f0 con 0x7efd94070f80 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 >> v1:172.21.15.77:6789/0 conn(0x7efd94070f80 legacy=0x7efd940cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.694 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd80003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd80003f70 0x7efd80004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd80004d90 0x7efd80005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ccfb0 con 0x7efd80003f70 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8000bba0 con 0x7efd80004d90 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8000bd60 con 0x7efd9415c640 2023-07-04T18:52:03.695 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd80003f70 0x7efd80004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99e82700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd80003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd9a683700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd80004d90 0x7efd80005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.696 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 -- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 msgr2=0x7efd80003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:03.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd9415c640 0x7efd80003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 -- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd80004d90 msgr2=0x7efd80005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:03.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd80004d90 0x7efd80005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.697 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.688+0000 7efd99681700 1 -- 172.21.15.77:0/1018485167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd800035b0 con 0x7efd80003f70 2023-07-04T18:52:03.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd99681700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd80003f70 0x7efd80004850 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd9415ce60 tx=0x7efd80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780939b76b3558ea server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000c660 con 0x7efd80003f70 2023-07-04T18:52:03.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9000c7c0 con 0x7efd80003f70 2023-07-04T18:52:03.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd98e80700 1 -- 172.21.15.77:0/1018485167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000cb70 con 0x7efd80003f70 2023-07-04T18:52:03.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 -- 172.21.15.77:0/1018485167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd80003f70 msgr2=0x7efd80004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/1018485167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd80003f70 0x7efd80004850 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd9415ce60 tx=0x7efd80005880 comp rx=0 tx=0).stop 2023-07-04T18:52:03.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9a683700 1 -- 172.21.15.77:0/1018485167 reap_dead start 2023-07-04T18:52:03.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 -- 172.21.15.77:0/1018485167 shutdown_connections 2023-07-04T18:52:03.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 -- 172.21.15.77:0/1018485167 >> 172.21.15.77:0/1018485167 conn(0x7efd9406bba0 msgr2=0x7efd940b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 -- 172.21.15.77:0/1018485167 shutdown_connections 2023-07-04T18:52:03.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.689+0000 7efd9c0e6700 1 -- 172.21.15.77:0/1018485167 wait complete. 2023-07-04T18:52:03.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 Processor -- start 2023-07-04T18:52:03.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 -- start start 2023-07-04T18:52:03.703 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd80003f70 0x7efd9415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd94153d70 0x7efd9415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94158d40 con 0x7efd940c7e20 2023-07-04T18:52:03.704 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940c76a0 con 0x7efd94153d70 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9c0e6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940c8800 con 0x7efd80003f70 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9a683700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9a683700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45400/0 (socket says 172.21.15.77:45400) 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.691+0000 7efd9a683700 1 -- 172.21.15.77:0/2637679635 learned_addr learned my addr 172.21.15.77:0/2637679635 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd99e82700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd80003f70 0x7efd9415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd99681700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd94153d70 0x7efd9415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.705 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 -- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd80003f70 msgr2=0x7efd9415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd80003f70 0x7efd9415b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 -- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd94153d70 msgr2=0x7efd9415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd94153d70 0x7efd9415be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 -- 172.21.15.77:0/2637679635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940c6a90 con 0x7efd940c7e20 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9a683700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efd940695f0 tx=0x7efd8c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00ecd0 con 0x7efd940c7e20 2023-07-04T18:52:03.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c00ee30 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940c6d70 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c013660 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.692+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd940c7330 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.693+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd940c74c0 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.693+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd8c0137c0 con 0x7efd940c7e20 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.694+0000 7efd8a7fc700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 0x7efd7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:03.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.694+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6120+0+0 (secure 0 0 0) 0x7efd8c089ac0 con 0x7efd940c7e20 2023-07-04T18:52:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.694+0000 7efd99e82700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 0x7efd7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.694+0000 7efd99e82700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 0x7efd7806d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd940ce210 tx=0x7efd84008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:03.708 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.698+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd8c054b30 con 0x7efd940c7e20 2023-07-04T18:52:03.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.899+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efd9404e210 con 0x7efd940c7e20 2023-07-04T18:52:03.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.899+0000 7efd8a7fc700 1 -- 172.21.15.77:0/2637679635 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7efd8c089e60 con 0x7efd940c7e20 2023-07-04T18:52:03.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.901+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 msgr2=0x7efd7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.901+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 0x7efd7806d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efd940ce210 tx=0x7efd84008040 comp rx=0 tx=0).stop 2023-07-04T18:52:03.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.901+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 msgr2=0x7efd940c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.901+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efd940695f0 tx=0x7efd8c00e440 comp rx=0 tx=0).stop 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 shutdown_connections 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efd7806ace0 0x7efd7806d190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd940c7e20 0x7efd940c82c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efd80003f70 0x7efd9415b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 --2- 172.21.15.77:0/2637679635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd94153d70 0x7efd9415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:03.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 >> 172.21.15.77:0/2637679635 conn(0x7efd9406bba0 msgr2=0x7efd9415a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:03.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 shutdown_connections 2023-07-04T18:52:03.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:03.902+0000 7efd9c0e6700 1 -- 172.21.15.77:0/2637679635 wait complete. 2023-07-04T18:52:03.905 INFO:teuthology.orchestra.run.smithi077.stderr:listed 0 entries 2023-07-04T18:52:03.916 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-07-04T18:52:03.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:03.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-07-04T18:52:03.969 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T18:52:03.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:03.970 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-07-04T18:52:03.991 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-07-04T18:52:03.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:04.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.111+0000 7f56a89a8700 1 Processor -- start 2023-07-04T18:52:04.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.111+0000 7f56a89a8700 1 -- start start 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.111+0000 7f56a89a8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56a415ca10 0x7f56a4070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a89a8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40cf830 con 0x7f56a4070db0 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a89a8700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40711d0 con 0x7f56a415c640 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a89a8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40717f0 con 0x7f56a415ca10 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a1d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56a415ca10 0x7f56a4070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a2d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f56a4070db0 0x7f56a40cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52546/0 (socket says 172.21.15.77:52546) 2023-07-04T18:52:04.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a2d9d700 1 -- 172.21.15.77:0/545161509 learned_addr learned my addr 172.21.15.77:0/545161509 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:04.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a1d9b700 1 -- 172.21.15.77:0/545161509 >> v1:172.21.15.82:6789/0 conn(0x7f56a415c640 legacy=0x7f56a40cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:04.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a1d9b700 1 -- 172.21.15.77:0/545161509 >> v1:172.21.15.77:6789/0 conn(0x7f56a4070db0 legacy=0x7f56a40cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a159a700 1 -- 172.21.15.77:0/545161509 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005554147 0 0) 0x7f56a40cf830 con 0x7f56a4070db0 2023-07-04T18:52:04.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a1d9b700 1 -- 172.21.15.77:0/545161509 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40cd290 con 0x7f56a415ca10 2023-07-04T18:52:04.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a1d9b700 1 --2- 172.21.15.77:0/545161509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a4070870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5694009840 tx=0x7f5694009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709c562f8d4cf3fc server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a159a700 1 -- 172.21.15.77:0/545161509 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694004230 con 0x7f56a415ca10 2023-07-04T18:52:04.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a159a700 1 -- 172.21.15.77:0/545161509 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694004390 con 0x7f56a415ca10 2023-07-04T18:52:04.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.112+0000 7f56a159a700 1 -- 172.21.15.77:0/545161509 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940036a0 con 0x7f56a415ca10 2023-07-04T18:52:04.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 -- 172.21.15.77:0/545161509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 msgr2=0x7f56a4070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 --2- 172.21.15.77:0/545161509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a4070870 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5694009840 tx=0x7f5694009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:04.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 -- 172.21.15.77:0/545161509 shutdown_connections 2023-07-04T18:52:04.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 --2- 172.21.15.77:0/545161509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a4070870 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 -- 172.21.15.77:0/545161509 >> 172.21.15.77:0/545161509 conn(0x7f56a406bba0 msgr2=0x7f56a40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:04.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 -- 172.21.15.77:0/545161509 shutdown_connections 2023-07-04T18:52:04.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.113+0000 7f56a89a8700 1 -- 172.21.15.77:0/545161509 wait complete. 2023-07-04T18:52:04.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.114+0000 7f56a89a8700 1 Processor -- start 2023-07-04T18:52:04.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.114+0000 7f56a89a8700 1 -- start start 2023-07-04T18:52:04.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 0x7f56a41f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 0x7f56a40c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40c9e30 con 0x7f56a415ca10 2023-07-04T18:52:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40c9870 con 0x7f56a4070db0 2023-07-04T18:52:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a89a8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40c9b50 con 0x7f56a415c640 2023-07-04T18:52:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45414/0 (socket says 172.21.15.77:45414) 2023-07-04T18:52:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 -- 172.21.15.77:0/215225719 learned_addr learned my addr 172.21.15.77:0/215225719 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a1d9b700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 0x7f56a40c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a259c700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 0x7f56a41f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 -- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 msgr2=0x7f56a40c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 0x7f56a40c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 -- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 msgr2=0x7f56a41f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 0x7f56a41f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a2d9d700 1 -- 172.21.15.77:0/215225719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940094a0 con 0x7f56a415ca10 2023-07-04T18:52:04.129 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a259c700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 0x7f56a41f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:04.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.115+0000 7f56a1d9b700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 0x7f56a40c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:04.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f56a2d9d700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f569c00cb60 tx=0x7f569c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c00ab50 con 0x7f56a415ca10 2023-07-04T18:52:04.130 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c00acb0 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a40b2d50 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c007310 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.116+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40c6cb0 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.117+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a40b2bb0 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.117+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f569c007470 con 0x7f56a415ca10 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.117+0000 7f5692ffd700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 0x7f568406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.131 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.118+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f569c089240 con 0x7f56a415ca10 2023-07-04T18:52:04.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.118+0000 7f56a259c700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 0x7f568406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.118+0000 7f56a259c700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 0x7f568406d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5698009930 tx=0x7f5698008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.132 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.122+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f56a40b2bb0 con 0x7f56a415ca10 2023-07-04T18:52:04.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.350+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56a40b3270 con 0x7f56a415ca10 2023-07-04T18:52:04.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.350+0000 7f5692ffd700 1 -- 172.21.15.77:0/215225719 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7f56a40b3270 con 0x7f56a415ca10 2023-07-04T18:52:04.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:04.353 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:52:04.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 msgr2=0x7f568406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 0x7f568406d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5698009930 tx=0x7f5698008040 comp rx=0 tx=0).stop 2023-07-04T18:52:04.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 msgr2=0x7f56a40cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.354 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f569c00cb60 tx=0x7f569c00cf20 comp rx=0 tx=0).stop 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 shutdown_connections 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f568406ace0 0x7f568406d190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56a415ca10 0x7f56a40cb6e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4070db0 0x7f56a41f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 --2- 172.21.15.77:0/215225719 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f56a415c640 0x7f56a40c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.352+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 >> 172.21.15.77:0/215225719 conn(0x7f56a406bba0 msgr2=0x7f56a41e3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:04.355 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.353+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 shutdown_connections 2023-07-04T18:52:04.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.353+0000 7f56a89a8700 1 -- 172.21.15.77:0/215225719 wait complete. 2023-07-04T18:52:04.356 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:52:04.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:04.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.485+0000 7f125dc7f700 1 Processor -- start 2023-07-04T18:52:04.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 -- start start 2023-07-04T18:52:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f125815c640 0x7f12580cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1258070f80 0x7f12580cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580717f0 con 0x7f125815ca10 2023-07-04T18:52:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12580cd290 con 0x7f125815c640 2023-07-04T18:52:04.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f125dc7f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12580ccfb0 con 0x7f1258070f80 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f1256ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f125815ca10 0x7f1258070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52556/0 (socket says 172.21.15.77:52556) 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f12577fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f125815c640 0x7f12580cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.486+0000 7f1256ffd700 1 -- 172.21.15.77:0/3969023204 learned_addr learned my addr 172.21.15.77:0/3969023204 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f1257fff700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1258070f80 0x7f12580cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12567fc700 1 -- 172.21.15.77:0/3969023204 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713988821 0 0) 0x7f12580717f0 con 0x7f125815ca10 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12567fc700 1 -- 172.21.15.77:0/3969023204 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12440035b0 con 0x7f125815ca10 2023-07-04T18:52:04.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12577fe700 1 -- 172.21.15.77:0/3969023204 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1258070f80 msgr2=0x7f12580cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:04.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12577fe700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1258070f80 0x7f12580cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12577fe700 1 -- 172.21.15.77:0/3969023204 >> v1:172.21.15.77:6789/0 conn(0x7f125815ca10 legacy=0x7f1258070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12577fe700 1 -- 172.21.15.77:0/3969023204 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580ce640 con 0x7f125815c640 2023-07-04T18:52:04.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12577fe700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f125815c640 0x7f12580cf120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1248009840 tx=0x7f1248009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a63d0ca14dcd80 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12567fc700 1 -- 172.21.15.77:0/3969023204 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248011830 con 0x7f125815c640 2023-07-04T18:52:04.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12567fc700 1 -- 172.21.15.77:0/3969023204 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248011990 con 0x7f125815c640 2023-07-04T18:52:04.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.487+0000 7f12567fc700 1 -- 172.21.15.77:0/3969023204 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248011d20 con 0x7f125815c640 2023-07-04T18:52:04.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 -- 172.21.15.77:0/3969023204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f125815c640 msgr2=0x7f12580cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f125815c640 0x7f12580cf120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1248009840 tx=0x7f1248009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 -- 172.21.15.77:0/3969023204 shutdown_connections 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1258070f80 0x7f12580cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 --2- 172.21.15.77:0/3969023204 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f125815c640 0x7f12580cf120 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 -- 172.21.15.77:0/3969023204 >> 172.21.15.77:0/3969023204 conn(0x7f125806bba0 msgr2=0x7f12580b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 -- 172.21.15.77:0/3969023204 shutdown_connections 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.488+0000 7f125dc7f700 1 -- 172.21.15.77:0/3969023204 wait complete. 2023-07-04T18:52:04.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.489+0000 7f125dc7f700 1 Processor -- start 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.489+0000 7f125dc7f700 1 -- start start 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 0x7f12581f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 0x7f12580b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12580b3fe0 con 0x7f1258070f80 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12580b5950 con 0x7f125815ca10 2023-07-04T18:52:04.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f125dc7f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12580b3740 con 0x7f125815c640 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45424/0 (socket says 172.21.15.77:45424) 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 -- 172.21.15.77:0/1649907263 learned_addr learned my addr 172.21.15.77:0/1649907263 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f1257fff700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 0x7f12580b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f1256ffd700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 0x7f12581f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 -- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 msgr2=0x7f12581f9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 0x7f12581f9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 -- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 msgr2=0x7f12580b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 0x7f12580b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 -- 172.21.15.77:0/1649907263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12480094a0 con 0x7f1258070f80 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f1257fff700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 0x7f12580b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f1256ffd700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 0x7f12581f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.490+0000 7f12577fe700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1248009fd0 tx=0x7f1248004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480036a0 con 0x7f1258070f80 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124801e070 con 0x7f1258070f80 2023-07-04T18:52:04.503 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12580b3a20 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248003a60 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1258157f00 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.491+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1258157d60 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.492+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1248003bc0 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.492+0000 7f124ffff700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 0x7f123806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.493+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f124808a480 con 0x7f1258070f80 2023-07-04T18:52:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.493+0000 7f1256ffd700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 0x7f123806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.493+0000 7f1256ffd700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 0x7f123806d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1258070460 tx=0x7f1240009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.505 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.497+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1258157d60 con 0x7f1258070f80 2023-07-04T18:52:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.690+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1258157d60 con 0x7f1258070f80 2023-07-04T18:52:04.692 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.691+0000 7f124ffff700 1 -- 172.21.15.77:0/1649907263 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18338 (secure 0 0 0) 0x7f1258157d60 con 0x7f1258070f80 2023-07-04T18:52:04.693 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:04.693 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:04.372702+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:04.698 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 msgr2=0x7f123806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 0x7f123806d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1258070460 tx=0x7f1240009040 comp rx=0 tx=0).stop 2023-07-04T18:52:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 msgr2=0x7f12580b5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1248009fd0 tx=0x7f1248004750 comp rx=0 tx=0).stop 2023-07-04T18:52:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 shutdown_connections 2023-07-04T18:52:04.699 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f123806ace0 0x7f123806d190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1258070f80 0x7f12580b5410 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f125815ca10 0x7f12580b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 --2- 172.21.15.77:0/1649907263 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f125815c640 0x7f12581f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.700 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 >> 172.21.15.77:0/1649907263 conn(0x7f125806bba0 msgr2=0x7f12581e6570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:04.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 shutdown_connections 2023-07-04T18:52:04.701 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.693+0000 7f125dc7f700 1 -- 172.21.15.77:0/1649907263 wait complete. 2023-07-04T18:52:04.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-04T18:51:20.418518+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-07-04T18:51:20.805906+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:52:04.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:52:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.837+0000 7f2b7961c700 1 Processor -- start 2023-07-04T18:52:04.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.837+0000 7f2b7961c700 1 -- start start 2023-07-04T18:52:04.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b7961c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b740b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b7961c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740ac8a0 con 0x7f2b7406ea90 2023-07-04T18:52:04.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b7961c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7415d0b0 con 0x7f2b7415d540 2023-07-04T18:52:04.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b7961c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7415ca80 con 0x7f2b7406a550 2023-07-04T18:52:04.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b72ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2b7406ea90 0x7f2b74160a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52570/0 (socket says 172.21.15.77:52570) 2023-07-04T18:52:04.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b72ffd700 1 -- 172.21.15.77:0/3908147026 learned_addr learned my addr 172.21.15.77:0/3908147026 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:04.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b737fe700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b740b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045131808 0 0) 0x7f2b740ac8a0 con 0x7f2b7406ea90 2023-07-04T18:52:04.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.838+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c0035b0 con 0x7f2b7406ea90 2023-07-04T18:52:04.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976150544 0 0) 0x7f2b5c0035b0 con 0x7f2b7406ea90 2023-07-04T18:52:04.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 msgr2=0x7f2b740b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:04.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b740b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 >> v1:172.21.15.82:6789/0 conn(0x7f2b7415d540 legacy=0x7f2b74069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740acaf0 con 0x7f2b7406ea90 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f2b68001670 con 0x7f2b7406ea90 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 >> v1:172.21.15.77:6789/0 conn(0x7f2b7406ea90 legacy=0x7f2b74160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b5c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c003f70 0x7f2b5c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c004d90 0x7f2b5c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7415ca80 con 0x7f2b5c003f70 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c00baf0 con 0x7f2b5c004d90 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c00bcb0 con 0x7f2b7406a550 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c003f70 0x7f2b5c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b72ffd700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b5c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b737fe700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c004d90 0x7f2b5c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 -- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 msgr2=0x7f2b5c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b7406a550 0x7f2b5c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 -- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c004d90 msgr2=0x7f2b5c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c004d90 0x7f2b5c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 -- 172.21.15.77:0/3908147026 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0035b0 con 0x7f2b5c003f70 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c003f70 0x7f2b5c004850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b64007fd0 tx=0x7f2b5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509cfb5f22039fbf server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400c660 con 0x7f2b5c003f70 2023-07-04T18:52:04.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.839+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6400c7c0 con 0x7f2b5c003f70 2023-07-04T18:52:04.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b71ffb700 1 -- 172.21.15.77:0/3908147026 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400cb70 con 0x7f2b5c003f70 2023-07-04T18:52:04.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 -- 172.21.15.77:0/3908147026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c003f70 msgr2=0x7f2b5c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3908147026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c003f70 0x7f2b5c004850 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b64007fd0 tx=0x7f2b5c005880 comp rx=0 tx=0).stop 2023-07-04T18:52:04.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b737fe700 1 -- 172.21.15.77:0/3908147026 reap_dead start 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 -- 172.21.15.77:0/3908147026 shutdown_connections 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 -- 172.21.15.77:0/3908147026 >> 172.21.15.77:0/3908147026 conn(0x7f2b7406af50 msgr2=0x7f2b7406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 -- 172.21.15.77:0/3908147026 shutdown_connections 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.840+0000 7f2b7961c700 1 -- 172.21.15.77:0/3908147026 wait complete. 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.841+0000 7f2b7961c700 1 Processor -- start 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.841+0000 7f2b7961c700 1 -- start start 2023-07-04T18:52:04.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 0x7f2b74153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b74155760 0x7f2b74155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741f1dc0 con 0x7f2b741505c0 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74156110 con 0x7f2b5c003f70 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b7961c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74150e60 con 0x7f2b74155760 2023-07-04T18:52:04.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45446/0 (socket says 172.21.15.77:45446) 2023-07-04T18:52:04.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 -- 172.21.15.77:0/3123799197 learned_addr learned my addr 172.21.15.77:0/3123799197 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:04.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b737fe700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b74155760 0x7f2b74155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b72ffd700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 0x7f2b74153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 -- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b74155760 msgr2=0x7f2b74155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b74155760 0x7f2b74155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 -- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 msgr2=0x7f2b74153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:04.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.842+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 0x7f2b74153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:04.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b727fc700 1 -- 172.21.15.77:0/3123799197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b741542d0 con 0x7f2b741505c0 2023-07-04T18:52:04.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b72ffd700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 0x7f2b74153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:04.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b727fc700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005300 tx=0x7f2b64001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400cb30 con 0x7f2b741505c0 2023-07-04T18:52:04.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64021070 con 0x7f2b741505c0 2023-07-04T18:52:04.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b741f1fd0 con 0x7f2b741505c0 2023-07-04T18:52:04.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64001330 con 0x7f2b741505c0 2023-07-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.843+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741f2590 con 0x7f2b741505c0 2023-07-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.844+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b740b4830 con 0x7f2b741505c0 2023-07-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.844+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2b64023030 con 0x7f2b741505c0 2023-07-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.845+0000 7f2b5b7fe700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 0x7f2b6006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:04.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.845+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f2b6408a4b0 con 0x7f2b741505c0 2023-07-04T18:52:04.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.845+0000 7f2b72ffd700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 0x7f2b6006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:04.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.845+0000 7f2b72ffd700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 0x7f2b6006d1c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b680011e0 tx=0x7f2b68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:04.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:04.849+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2b64055690 con 0x7f2b741505c0 2023-07-04T18:52:05.036 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.034+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b7404e210 con 0x7f2b741505c0 2023-07-04T18:52:05.036 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.035+0000 7f2b5b7fe700 1 -- 172.21.15.77:0/3123799197 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b64058cb0 con 0x7f2b741505c0 2023-07-04T18:52:05.037 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:52:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 msgr2=0x7f2b6006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 0x7f2b6006d1c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b680011e0 tx=0x7f2b68009040 comp rx=0 tx=0).stop 2023-07-04T18:52:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 msgr2=0x7f2b74143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005300 tx=0x7f2b64001230 comp rx=0 tx=0).stop 2023-07-04T18:52:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 shutdown_connections 2023-07-04T18:52:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2b6006ad10 0x7f2b6006d1c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2b74155760 0x7f2b74155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b5c003f70 0x7f2b74153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 --2- 172.21.15.77:0/3123799197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b741505c0 0x7f2b74143430 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 >> 172.21.15.77:0/3123799197 conn(0x7f2b7406af50 msgr2=0x7f2b740af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 shutdown_connections 2023-07-04T18:52:05.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.037+0000 7f2b7961c700 1 -- 172.21.15.77:0/3123799197 wait complete. 2023-07-04T18:52:05.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:52:05.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.172+0000 7f099c882700 1 Processor -- start 2023-07-04T18:52:05.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.172+0000 7f099c882700 1 -- start start 2023-07-04T18:52:05.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099c882700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 0x7f099806a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099c882700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 0x7f099815db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099c882700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09980cf750 con 0x7f09980d01c0 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099c882700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09980af060 con 0x7f0998069e40 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099c882700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09980cf9c0 con 0x7f09980aeb40 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099659c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 0x7f099806a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f0996d9d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 0x7f099815db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099659c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 0x7f099806a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:51258/0 (socket says 172.21.15.77:51258) 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f099659c700 1 -- 172.21.15.77:0/381911568 learned_addr learned my addr 172.21.15.77:0/381911568 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.173+0000 7f0995d9b700 1 --1- 172.21.15.77:0/381911568 >> v1:172.21.15.82:6789/0 conn(0x7f09980d01c0 0x7f09980ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60502/0 (socket says 172.21.15.77:60502) 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f0996d9d700 1 -- 172.21.15.77:0/381911568 >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 msgr2=0x7f099806a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f0996d9d700 1 --2- 172.21.15.77:0/381911568 >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 0x7f099806a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f0996d9d700 1 -- 172.21.15.77:0/381911568 >> v1:172.21.15.82:6789/0 conn(0x7f09980d01c0 legacy=0x7f09980ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f0996d9d700 1 -- 172.21.15.77:0/381911568 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09980cfca0 con 0x7f09980aeb40 2023-07-04T18:52:05.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f0996d9d700 1 --2- 172.21.15.77:0/381911568 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 0x7f099815db70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b518c994086a58d server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f099559a700 1 -- 172.21.15.77:0/381911568 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099000e070 con 0x7f09980aeb40 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f099559a700 1 -- 172.21.15.77:0/381911568 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099000bb20 con 0x7f09980aeb40 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.174+0000 7f099559a700 1 -- 172.21.15.77:0/381911568 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990012430 con 0x7f09980aeb40 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 -- 172.21.15.77:0/381911568 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 msgr2=0x7f099815db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 --2- 172.21.15.77:0/381911568 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 0x7f099815db70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099000b330 comp rx=0 tx=0).stop 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 -- 172.21.15.77:0/381911568 shutdown_connections 2023-07-04T18:52:05.180 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 --2- 172.21.15.77:0/381911568 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980aeb40 0x7f099815db70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 --2- 172.21.15.77:0/381911568 >> v2:172.21.15.82:3300/0 conn(0x7f0998069e40 0x7f099806a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 -- 172.21.15.77:0/381911568 >> 172.21.15.77:0/381911568 conn(0x7f099806af50 msgr2=0x7f099806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 -- 172.21.15.77:0/381911568 shutdown_connections 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.175+0000 7f099c882700 1 -- 172.21.15.77:0/381911568 wait complete. 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.176+0000 7f099c882700 1 Processor -- start 2023-07-04T18:52:05.181 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.176+0000 7f099c882700 1 -- start start 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09980aeb40 0x7f09980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980d01c0 0x7f09980af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998157860 con 0x7f0998069e40 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980b2540 con 0x7f09980aeb40 2023-07-04T18:52:05.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099815c010 con 0x7f09980d01c0 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45456/0 (socket says 172.21.15.77:45456) 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 -- 172.21.15.77:0/1739839359 learned_addr learned my addr 172.21.15.77:0/1739839359 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f0996d9d700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980d01c0 0x7f09980af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f0995d9b700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09980aeb40 0x7f09980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 -- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980d01c0 msgr2=0x7f09980af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980d01c0 0x7f09980af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 -- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09980aeb40 msgr2=0x7f09980b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09980aeb40 0x7f09980b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 -- 172.21.15.77:0/1739839359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990009a70 con 0x7f0998069e40 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099659c700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f098c00e8b0 tx=0x7f098c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c009600 con 0x7f0998069e40 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c013680 con 0x7f0998069e40 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09981571e0 con 0x7f0998069e40 2023-07-04T18:52:05.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.177+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c013a30 con 0x7f0998069e40 2023-07-04T18:52:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.178+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0998157480 con 0x7f0998069e40 2023-07-04T18:52:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.178+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0998156b30 con 0x7f0998069e40 2023-07-04T18:52:05.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.179+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f098c013b90 con 0x7f0998069e40 2023-07-04T18:52:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.179+0000 7f0986ffd700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 0x7f097806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.179+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f098c0892f0 con 0x7f0998069e40 2023-07-04T18:52:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.179+0000 7f0995d9b700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 0x7f097806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.180+0000 7f0995d9b700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 0x7f097806d190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f099806e780 tx=0x7f0988007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.184+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f098c057e00 con 0x7f0998069e40 2023-07-04T18:52:05.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.369+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f099804e210 con 0x7f0998069e40 2023-07-04T18:52:05.371 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.369+0000 7f0986ffd700 1 -- 172.21.15.77:0/1739839359 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7f098c057990 con 0x7f0998069e40 2023-07-04T18:52:05.371 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:52:05.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.371+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 msgr2=0x7f097806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.371+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 0x7f097806d190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f099806e780 tx=0x7f0988007ed0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.371+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 msgr2=0x7f099815b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.371+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f098c00e8b0 tx=0x7f098c00ebc0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 shutdown_connections 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f097806ace0 0x7f097806d190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0998069e40 0x7f099815b650 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f09980d01c0 0x7f09980af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 --2- 172.21.15.77:0/1739839359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09980aeb40 0x7f09980b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 >> 172.21.15.77:0/1739839359 conn(0x7f099806af50 msgr2=0x7f09980b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 shutdown_connections 2023-07-04T18:52:05.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.372+0000 7f099c882700 1 -- 172.21.15.77:0/1739839359 wait complete. 2023-07-04T18:52:05.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-04T18:52:05.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.506+0000 7fc112162700 1 Processor -- start 2023-07-04T18:52:05.508 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 -- start start 2023-07-04T18:52:05.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fc10c069e40 0x7fc10c06a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c0d01c0 0x7fc10c0ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.510 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc10c0ae970 0x7fc10c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10c0cf750 con 0x7fc10c0ae970 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10c0aedd0 con 0x7fc10c069e40 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc112162700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c0af060 con 0x7fc10c0d01c0 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.507+0000 7fc10bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc10c0ae970 0x7fc10c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc10c0ae970 0x7fc10c15db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45468/0 (socket says 172.21.15.77:45468) 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 -- 172.21.15.77:0/1537092964 learned_addr learned my addr 172.21.15.77:0/1537092964 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10b7fe700 1 --2- 172.21.15.77:0/1537092964 >> v2:172.21.15.82:3300/0 conn(0x7fc10c069e40 0x7fc10c06a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10affd700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c0d01c0 0x7fc10c0ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 -- 172.21.15.77:0/1537092964 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c0d01c0 msgr2=0x7fc10c0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c0d01c0 0x7fc10c0ae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 -- 172.21.15.77:0/1537092964 >> v2:172.21.15.82:3300/0 conn(0x7fc10c069e40 msgr2=0x7fc10c06a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 --2- 172.21.15.77:0/1537092964 >> v2:172.21.15.82:3300/0 conn(0x7fc10c069e40 0x7fc10c06a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 -- 172.21.15.77:0/1537092964 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c0cf9c0 con 0x7fc10c0ae970 2023-07-04T18:52:05.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10bfff700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c15db70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0712a0 tx=0x7fc10400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da05c46e7932c391 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10a7fc700 1 -- 172.21.15.77:0/1537092964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10400e070 con 0x7fc10c0ae970 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10a7fc700 1 -- 172.21.15.77:0/1537092964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10400b910 con 0x7fc10c0ae970 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.508+0000 7fc10a7fc700 1 -- 172.21.15.77:0/1537092964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10400bcc0 con 0x7fc10c0ae970 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 -- 172.21.15.77:0/1537092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 msgr2=0x7fc10c15db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c15db70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0712a0 tx=0x7fc10400b330 comp rx=0 tx=0).stop 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 -- 172.21.15.77:0/1537092964 shutdown_connections 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c15db70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.513 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 --2- 172.21.15.77:0/1537092964 >> v2:172.21.15.82:3300/0 conn(0x7fc10c069e40 0x7fc10c06a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 --2- 172.21.15.77:0/1537092964 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c0d01c0 0x7fc10c0ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 -- 172.21.15.77:0/1537092964 >> 172.21.15.77:0/1537092964 conn(0x7fc10c06af50 msgr2=0x7fc10c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 -- 172.21.15.77:0/1537092964 shutdown_connections 2023-07-04T18:52:05.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.509+0000 7fc112162700 1 -- 172.21.15.77:0/1537092964 wait complete. 2023-07-04T18:52:05.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.510+0000 7fc112162700 1 Processor -- start 2023-07-04T18:52:05.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.510+0000 7fc112162700 1 -- start start 2023-07-04T18:52:05.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c069e40 0x7fc10c156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc10c0d01c0 0x7fc10c143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c143f80 con 0x7fc10c0ae970 2023-07-04T18:52:05.517 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c15c020 con 0x7fc10c0d01c0 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc112162700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c15b260 con 0x7fc10c069e40 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45472/0 (socket says 172.21.15.77:45472) 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 -- 172.21.15.77:0/1708812797 learned_addr learned my addr 172.21.15.77:0/1708812797 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10b7fe700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c069e40 0x7fc10c156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.518 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10bfff700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc10c0d01c0 0x7fc10c143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 -- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c069e40 msgr2=0x7fc10c156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.519 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c069e40 0x7fc10c156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 -- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc10c0d01c0 msgr2=0x7fc10c143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc10c0d01c0 0x7fc10c143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 -- 172.21.15.77:0/1708812797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc104009b30 con 0x7fc10c0ae970 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc10affd700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00e9b0 tx=0x7fc0fc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.511+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc009710 con 0x7fc10c0ae970 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.512+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0fc00c850 con 0x7fc10c0ae970 2023-07-04T18:52:05.520 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.512+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c15b540 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.512+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc00cbc0 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.512+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c15bb00 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.512+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c15bcc0 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.513+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc0fc00cd20 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.513+0000 7fc103fff700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 0x7fc0ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.514+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fc0fc089540 con 0x7fc10c0ae970 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.514+0000 7fc10b7fe700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 0x7fc0ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.521 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.514+0000 7fc10b7fe700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 0x7fc0ec06d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009a50 tx=0x7fc0f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.522 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.518+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc0fc0580d0 con 0x7fc10c0ae970 2023-07-04T18:52:05.706 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.705+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc10c04e210 con 0x7fc10c0ae970 2023-07-04T18:52:05.707 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.705+0000 7fc103fff700 1 -- 172.21.15.77:0/1708812797 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0fc057c60 con 0x7fc10c0ae970 2023-07-04T18:52:05.707 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2023-07-04T18:52:05.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 msgr2=0x7fc0ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 0x7fc0ec06d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009a50 tx=0x7fc0f4008040 comp rx=0 tx=0).stop 2023-07-04T18:52:05.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 msgr2=0x7fc10c143500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00e9b0 tx=0x7fc0fc00ed70 comp rx=0 tx=0).stop 2023-07-04T18:52:05.709 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 shutdown_connections 2023-07-04T18:52:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc0ec06ace0 0x7fc0ec06d190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc10c0d01c0 0x7fc10c143a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc10c069e40 0x7fc10c156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 --2- 172.21.15.77:0/1708812797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc10c0ae970 0x7fc10c143500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.710 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 >> 172.21.15.77:0/1708812797 conn(0x7fc10c06af50 msgr2=0x7fc10c0ad7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 shutdown_connections 2023-07-04T18:52:05.711 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.707+0000 7fc112162700 1 -- 172.21.15.77:0/1708812797 wait complete. 2023-07-04T18:52:05.721 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-04T18:52:05.845 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.844+0000 7fbf771a9700 1 Processor -- start 2023-07-04T18:52:05.846 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.844+0000 7fbf771a9700 1 -- start start 2023-07-04T18:52:05.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.844+0000 7fbf771a9700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 0x7fbf7015a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.844+0000 7fbf771a9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf771a9700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf701582f0 con 0x7fbf70069e40 2023-07-04T18:52:05.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf771a9700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7015de50 con 0x7fbf70157f20 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf771a9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015b0b0 con 0x7fbf7015e580 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf74f45700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 0x7fbf7015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:35328/0 (socket says 172.21.15.77:35328) 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf74f45700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 0x7fbf7015a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:51284/0 (socket says 172.21.15.77:51284) 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 -- 172.21.15.77:0/2049285702 learned_addr learned my addr 172.21.15.77:0/2049285702 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 -- 172.21.15.77:0/2049285702 >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 msgr2=0x7fbf7015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:05.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/2049285702 >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 0x7fbf7015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 -- 172.21.15.77:0/2049285702 >> v1:172.21.15.82:6789/0 conn(0x7fbf70069e40 legacy=0x7fbf700cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 -- 172.21.15.77:0/2049285702 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7015b390 con 0x7fbf7015e580 2023-07-04T18:52:05.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.845+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/2049285702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0098f0 tx=0x7fbf5c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40c8df39847fc79 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf6f7fe700 1 -- 172.21.15.77:0/2049285702 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c011830 con 0x7fbf7015e580 2023-07-04T18:52:05.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf6f7fe700 1 -- 172.21.15.77:0/2049285702 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c011990 con 0x7fbf7015e580 2023-07-04T18:52:05.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf6f7fe700 1 -- 172.21.15.77:0/2049285702 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c011d20 con 0x7fbf7015e580 2023-07-04T18:52:05.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 -- 172.21.15.77:0/2049285702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 msgr2=0x7fbf7015e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 --2- 172.21.15.77:0/2049285702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0098f0 tx=0x7fbf5c009c00 comp rx=0 tx=0).stop 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 -- 172.21.15.77:0/2049285702 shutdown_connections 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 --2- 172.21.15.77:0/2049285702 >> v2:172.21.15.82:3300/0 conn(0x7fbf70157f20 0x7fbf7015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 --2- 172.21.15.77:0/2049285702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf7015e580 0x7fbf7015e9d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 -- 172.21.15.77:0/2049285702 >> 172.21.15.77:0/2049285702 conn(0x7fbf7006af50 msgr2=0x7fbf7006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 -- 172.21.15.77:0/2049285702 shutdown_connections 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.846+0000 7fbf771a9700 1 -- 172.21.15.77:0/2049285702 wait complete. 2023-07-04T18:52:05.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.847+0000 7fbf771a9700 1 Processor -- start 2023-07-04T18:52:05.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.847+0000 7fbf771a9700 1 -- start start 2023-07-04T18:52:05.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 0x7fbf7014c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf7015e580 0x7fbf7015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7014cbb0 con 0x7fbf70069e40 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015cd00 con 0x7fbf7015e580 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf771a9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7015cfd0 con 0x7fbf70157f20 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf74f45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf74f45700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45488/0 (socket says 172.21.15.77:45488) 2023-07-04T18:52:05.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf74f45700 1 -- 172.21.15.77:0/3330554594 learned_addr learned my addr 172.21.15.77:0/3330554594 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.848+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 0x7fbf7014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf75746700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf7015e580 0x7fbf7015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 -- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 msgr2=0x7fbf7014c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 0x7fbf7014c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 -- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf7015e580 msgr2=0x7fbf7015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:05.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf7015e580 0x7fbf7015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 -- 172.21.15.77:0/3330554594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c009550 con 0x7fbf70069e40 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 0x7fbf7014c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf74f45700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400e9b0 tx=0x7fbf6400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64009710 con 0x7fbf70069e40 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6400c850 con 0x7fbf70069e40 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7015d2b0 con 0x7fbf70069e40 2023-07-04T18:52:05.861 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400cbc0 con 0x7fbf70069e40 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.849+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7015d870 con 0x7fbf70069e40 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.850+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf700ad3d0 con 0x7fbf70069e40 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.851+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbf6400cd20 con 0x7fbf70069e40 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.851+0000 7fbf6d7fa700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 0x7fbf5406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.851+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fbf64089550 con 0x7fbf70069e40 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.851+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 0x7fbf5406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:05.862 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.852+0000 7fbf6ffff700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 0x7fbf5406d1e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c014040 tx=0x7fbf5c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:05.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:05.855+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbf64058130 con 0x7fbf70069e40 2023-07-04T18:52:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.041+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf7004e210 con 0x7fbf70069e40 2023-07-04T18:52:06.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.041+0000 7fbf6d7fa700 1 -- 172.21.15.77:0/3330554594 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf64057cc0 con 0x7fbf70069e40 2023-07-04T18:52:06.043 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2023-07-04T18:52:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 msgr2=0x7fbf5406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 0x7fbf5406d1e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c014040 tx=0x7fbf5c008030 comp rx=0 tx=0).stop 2023-07-04T18:52:06.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 msgr2=0x7fbf700b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400e9b0 tx=0x7fbf6400ed70 comp rx=0 tx=0).stop 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 shutdown_connections 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbf5406ad30 0x7fbf5406d1e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf70069e40 0x7fbf700b4050 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf7015e580 0x7fbf7015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 --2- 172.21.15.77:0/3330554594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbf70157f20 0x7fbf7014c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 >> 172.21.15.77:0/3330554594 conn(0x7fbf7006af50 msgr2=0x7fbf7006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 shutdown_connections 2023-07-04T18:52:06.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.043+0000 7fbf771a9700 1 -- 172.21.15.77:0/3330554594 wait complete. 2023-07-04T18:52:06.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:06.182 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.179+0000 7facf8b84700 1 Processor -- start 2023-07-04T18:52:06.182 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.179+0000 7facf8b84700 1 -- start start 2023-07-04T18:52:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf8b84700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7facf415a430 0x7facf40ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf8b84700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7facf40cebf0 0x7facf40cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.184 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf8b84700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40cf100 con 0x7facf415ea40 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf8b84700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf415df60 con 0x7facf415a430 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf8b84700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf415e240 con 0x7facf40cebf0 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7facf415a430 0x7facf40ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7facf415a430 0x7facf40ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45492/0 (socket says 172.21.15.77:45492) 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 -- 172.21.15.77:0/2882237143 learned_addr learned my addr 172.21.15.77:0/2882237143 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf1d9b700 1 --2- 172.21.15.77:0/2882237143 >> v2:172.21.15.82:3300/0 conn(0x7facf40cebf0 0x7facf40cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.185 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 -- 172.21.15.77:0/2882237143 >> v2:172.21.15.82:3300/0 conn(0x7facf40cebf0 msgr2=0x7facf40cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 --2- 172.21.15.77:0/2882237143 >> v2:172.21.15.82:3300/0 conn(0x7facf40cebf0 0x7facf40cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 -- 172.21.15.77:0/2882237143 >> v1:172.21.15.82:6789/0 conn(0x7facf415ea40 legacy=0x7facf40cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.180+0000 7facf259c700 1 -- 172.21.15.77:0/2882237143 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf415e520 con 0x7facf415a430 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf259c700 1 --2- 172.21.15.77:0/2882237143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf415a430 0x7facf40ce6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7facf40713a0 tx=0x7face8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8d28c7dcaf841f server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf159a700 1 -- 172.21.15.77:0/2882237143 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011830 con 0x7facf415a430 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf159a700 1 -- 172.21.15.77:0/2882237143 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8011990 con 0x7facf415a430 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf159a700 1 -- 172.21.15.77:0/2882237143 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011d20 con 0x7facf415a430 2023-07-04T18:52:06.186 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 -- 172.21.15.77:0/2882237143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf415a430 msgr2=0x7facf40ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 --2- 172.21.15.77:0/2882237143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf415a430 0x7facf40ce6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7facf40713a0 tx=0x7face8009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 -- 172.21.15.77:0/2882237143 shutdown_connections 2023-07-04T18:52:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 --2- 172.21.15.77:0/2882237143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf415a430 0x7facf40ce6b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.187 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 --2- 172.21.15.77:0/2882237143 >> v2:172.21.15.82:3300/0 conn(0x7facf40cebf0 0x7facf40cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 -- 172.21.15.77:0/2882237143 >> 172.21.15.77:0/2882237143 conn(0x7facf406d7c0 msgr2=0x7facf406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 -- 172.21.15.77:0/2882237143 shutdown_connections 2023-07-04T18:52:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.181+0000 7facf8b84700 1 -- 172.21.15.77:0/2882237143 wait complete. 2023-07-04T18:52:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 Processor -- start 2023-07-04T18:52:06.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 -- start start 2023-07-04T18:52:06.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facf415ea40 0x7facf40c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.195 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7facf40acfc0 0x7facf40ad440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40ade60 con 0x7facf40cebf0 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40b0df0 con 0x7facf415ea40 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf8b84700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40b10d0 con 0x7facf40acfc0 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45498/0 (socket says 172.21.15.77:45498) 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.183+0000 7facf259c700 1 -- 172.21.15.77:0/3366817175 learned_addr learned my addr 172.21.15.77:0/3366817175 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:06.196 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf1d9b700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facf415ea40 0x7facf40c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf2d9d700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7facf40acfc0 0x7facf40ad440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 -- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7facf40acfc0 msgr2=0x7facf40ad440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7facf40acfc0 0x7facf40ad440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 -- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facf415ea40 msgr2=0x7facf40c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facf415ea40 0x7facf40c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 -- 172.21.15.77:0/3366817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8009550 con 0x7facf40cebf0 2023-07-04T18:52:06.197 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf259c700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7face8000c00 tx=0x7face8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8004570 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face801f070 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf40b13b0 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face80038e0 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.184+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf41520a0 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.185+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf404e210 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.185+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7face8021030 con 0x7facf40cebf0 2023-07-04T18:52:06.198 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.186+0000 7face2ffd700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 0x7facd406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.186+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7face808a910 con 0x7facf40cebf0 2023-07-04T18:52:06.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.186+0000 7facf1d9b700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 0x7facd406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.186+0000 7facf1d9b700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 0x7facd406d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7facf40cde50 tx=0x7face4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.199 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.190+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7face8055a10 con 0x7facf40cebf0 2023-07-04T18:52:06.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.423+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facf40ce270 con 0x7facf40cebf0 2023-07-04T18:52:06.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.424+0000 7face2ffd700 1 -- 172.21.15.77:0/3366817175 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7face8015090 con 0x7facf40cebf0 2023-07-04T18:52:06.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:06.426 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:52:06.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 msgr2=0x7facd406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 0x7facd406d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7facf40cde50 tx=0x7face4008040 comp rx=0 tx=0).stop 2023-07-04T18:52:06.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 msgr2=0x7facf4151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7face8000c00 tx=0x7face8004750 comp rx=0 tx=0).stop 2023-07-04T18:52:06.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 shutdown_connections 2023-07-04T18:52:06.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7facd406ace0 0x7facd406d190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7facf40cebf0 0x7facf4151950 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7facf40acfc0 0x7facf40ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 --2- 172.21.15.77:0/3366817175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facf415ea40 0x7facf40c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 >> 172.21.15.77:0/3366817175 conn(0x7facf406d7c0 msgr2=0x7facf4155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 shutdown_connections 2023-07-04T18:52:06.429 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.426+0000 7facf8b84700 1 -- 172.21.15.77:0/3366817175 wait complete. 2023-07-04T18:52:06.430 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:52:06.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:06.560 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.559+0000 7f5d995be700 1 Processor -- start 2023-07-04T18:52:06.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.559+0000 7f5d995be700 1 -- start start 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d995be700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d940d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d995be700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d941610f0 con 0x7f5d94146270 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d995be700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d940b2790 con 0x7f5d940af8f0 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d995be700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940b2a70 con 0x7f5d940b1860 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d937fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5d94146270 0x7f5d941609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52648/0 (socket says 172.21.15.77:52648) 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d937fe700 1 -- 172.21.15.77:0/2602218632 learned_addr learned my addr 172.21.15.77:0/2602218632 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:06.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d92ffd700 1 --2- 172.21.15.77:0/2602218632 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d940d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454733926 0 0) 0x7f5d941610f0 con 0x7f5d94146270 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7c0035b0 con 0x7f5d94146270 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673697410 0 0) 0x7f5d940b2790 con 0x7f5d940af8f0 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d941610f0 con 0x7f5d940af8f0 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d92ffd700 1 -- 172.21.15.77:0/2602218632 >> v1:172.21.15.82:6789/0 conn(0x7f5d940af8f0 legacy=0x7f5d941602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d92ffd700 1 -- 172.21.15.77:0/2602218632 >> v1:172.21.15.77:6789/0 conn(0x7f5d94146270 legacy=0x7f5d941609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1999132965 0 0) 0x7f5d7c0035b0 con 0x7f5d94146270 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.560+0000 7f5d92ffd700 1 -- 172.21.15.77:0/2602218632 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940b2d50 con 0x7f5d940b1860 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d92ffd700 1 --2- 172.21.15.77:0/2602218632 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d940d2cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009870 tx=0x7f5d84009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378a1cd43aadcf32 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0019f0 con 0x7f5d940b1860 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d84004030 con 0x7f5d940b1860 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d91ffb700 1 -- 172.21.15.77:0/2602218632 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d840043c0 con 0x7f5d940b1860 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 -- 172.21.15.77:0/2602218632 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 msgr2=0x7f5d940d2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 --2- 172.21.15.77:0/2602218632 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d940d2cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009870 tx=0x7f5d84009b80 comp rx=0 tx=0).stop 2023-07-04T18:52:06.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 -- 172.21.15.77:0/2602218632 shutdown_connections 2023-07-04T18:52:06.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 --2- 172.21.15.77:0/2602218632 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d940d2cb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 -- 172.21.15.77:0/2602218632 >> 172.21.15.77:0/2602218632 conn(0x7f5d94071e90 msgr2=0x7f5d9409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 -- 172.21.15.77:0/2602218632 shutdown_connections 2023-07-04T18:52:06.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.561+0000 7f5d995be700 1 -- 172.21.15.77:0/2602218632 wait complete. 2023-07-04T18:52:06.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 Processor -- start 2023-07-04T18:52:06.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 -- start start 2023-07-04T18:52:06.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d94154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d94146270 0x7f5d940c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ca3a0 con 0x7f5d940af8f0 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940cd3c0 con 0x7f5d94146270 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d995be700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ca590 con 0x7f5d940b1860 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d92ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d92ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45512/0 (socket says 172.21.15.77:45512) 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.563+0000 7f5d92ffd700 1 -- 172.21.15.77:0/1165267702 learned_addr learned my addr 172.21.15.77:0/1165267702 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d927fc700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d94154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d937fe700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d94146270 0x7f5d940c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 -- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 msgr2=0x7f5d94154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d94154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 -- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d94146270 msgr2=0x7f5d940c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d94146270 0x7f5d940c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 -- 172.21.15.77:0/1165267702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d840094a0 con 0x7f5d940af8f0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d927fc700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d94154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d92ffd700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5d840090b0 tx=0x7f5d84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84004570 con 0x7f5d940af8f0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8401f070 con 0x7f5d940af8f0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d940ca870 con 0x7f5d940af8f0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84011870 con 0x7f5d940af8f0 2023-07-04T18:52:06.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.564+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d940cae30 con 0x7f5d940af8f0 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.565+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d94066fa0 con 0x7f5d940af8f0 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.565+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d840119d0 con 0x7f5d940af8f0 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.566+0000 7f5d7b7fe700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 0x7f5d8006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.566+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f5d84021070 con 0x7f5d940af8f0 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.566+0000 7f5d927fc700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 0x7f5d8006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.566+0000 7f5d927fc700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 0x7f5d8006d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5d880011e0 tx=0x7f5d88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.570+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d84055ee0 con 0x7f5d940af8f0 2023-07-04T18:52:06.759 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.758+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d940cd150 con 0x7f5d940af8f0 2023-07-04T18:52:06.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.759+0000 7f5d7b7fe700 1 -- 172.21.15.77:0/1165267702 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18338 (secure 0 0 0) 0x7f5d84059500 con 0x7f5d940af8f0 2023-07-04T18:52:06.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:04.372702+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 msgr2=0x7f5d8006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 0x7f5d8006d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5d880011e0 tx=0x7f5d88009040 comp rx=0 tx=0).stop 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 msgr2=0x7f5d940c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5d840090b0 tx=0x7f5d84004750 comp rx=0 tx=0).stop 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 shutdown_connections 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d8006ace0 0x7f5d8006d190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d940af8f0 0x7f5d940c9360 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d94146270 0x7f5d940c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 --2- 172.21.15.77:0/1165267702 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d940b1860 0x7f5d94154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 >> 172.21.15.77:0/1165267702 conn(0x7f5d94071e90 msgr2=0x7f5d940b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 shutdown_connections 2023-07-04T18:52:06.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.761+0000 7f5d995be700 1 -- 172.21.15.77:0/1165267702 wait complete. 2023-07-04T18:52:06.775 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-04T18:52:06.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:06.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.893+0000 7f8461fcd700 1 Processor -- start 2023-07-04T18:52:06.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.893+0000 7f8461fcd700 1 -- start start 2023-07-04T18:52:06.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f8461fcd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f845c159280 0x7f845c06c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f8461fcd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c0cdca0 0x7f845c0ca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f8461fcd700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c06d890 con 0x7f845c06c770 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f8461fcd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c06cc80 con 0x7f845c159280 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f8461fcd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c06cf60 con 0x7f845c0cdca0 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f845b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f845c159280 0x7f845c06c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f845b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f845c159280 0x7f845c06c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45520/0 (socket says 172.21.15.77:45520) 2023-07-04T18:52:06.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.894+0000 7f845b7fe700 1 -- 172.21.15.77:0/3639467691 learned_addr learned my addr 172.21.15.77:0/3639467691 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845bfff700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c0cdca0 0x7f845c0ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845b7fe700 1 -- 172.21.15.77:0/3639467691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c0cdca0 msgr2=0x7f845c0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845b7fe700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c0cdca0 0x7f845c0ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845b7fe700 1 -- 172.21.15.77:0/3639467691 >> v1:172.21.15.82:6789/0 conn(0x7f845c06c770 legacy=0x7f845c0cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845b7fe700 1 -- 172.21.15.77:0/3639467691 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c06d240 con 0x7f845c159280 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845b7fe700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c159280 0x7f845c06c230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f844400bb90 tx=0x7f844400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85d6b8e590e711e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845a7fc700 1 -- 172.21.15.77:0/3639467691 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400dd50 con 0x7f845c159280 2023-07-04T18:52:06.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845a7fc700 1 -- 172.21.15.77:0/3639467691 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444010470 con 0x7f845c159280 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.895+0000 7f845a7fc700 1 -- 172.21.15.77:0/3639467691 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444010800 con 0x7f845c159280 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 -- 172.21.15.77:0/3639467691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c159280 msgr2=0x7f845c06c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c159280 0x7f845c06c230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f844400bb90 tx=0x7f844400bea0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 -- 172.21.15.77:0/3639467691 shutdown_connections 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c0cdca0 0x7f845c0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 --2- 172.21.15.77:0/3639467691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c159280 0x7f845c06c230 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 -- 172.21.15.77:0/3639467691 >> 172.21.15.77:0/3639467691 conn(0x7f845c06bba0 msgr2=0x7f845c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:06.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 -- 172.21.15.77:0/3639467691 shutdown_connections 2023-07-04T18:52:06.901 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.896+0000 7f8461fcd700 1 -- 172.21.15.77:0/3639467691 wait complete. 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.897+0000 7f8461fcd700 1 Processor -- start 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.897+0000 7f8461fcd700 1 -- start start 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f845c0cdca0 0x7f845c1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c159280 0x7f845c14dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c14e250 con 0x7f845c06c770 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0cc7a0 con 0x7f845c0cdca0 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c0cbb90 con 0x7f845c159280 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45526/0 (socket says 172.21.15.77:45526) 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 -- 172.21.15.77:0/2071913895 learned_addr learned my addr 172.21.15.77:0/2071913895 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:06.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845affd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f845c0cdca0 0x7f845c1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845bfff700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c159280 0x7f845c14dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 -- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c159280 msgr2=0x7f845c14dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c159280 0x7f845c14dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 -- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f845c0cdca0 msgr2=0x7f845c1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f845c0cdca0 0x7f845c1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:06.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 -- 172.21.15.77:0/2071913895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844400b7f0 con 0x7f845c06c770 2023-07-04T18:52:06.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f845b7fe700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f844400bee0 tx=0x7f8444004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444003ab0 con 0x7f845c06c770 2023-07-04T18:52:06.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444003c30 con 0x7f845c06c770 2023-07-04T18:52:06.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440105e0 con 0x7f845c06c770 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.898+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c0cbe70 con 0x7f845c06c770 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.899+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c0cc430 con 0x7f845c06c770 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.899+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c0cc5c0 con 0x7f845c06c770 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.900+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8444019400 con 0x7f845c06c770 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.900+0000 7f8453fff700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 0x7f843c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:06.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.900+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f844408a7d0 con 0x7f845c06c770 2023-07-04T18:52:06.908 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.900+0000 7f845affd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 0x7f843c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:06.909 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.901+0000 7f845affd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 0x7f843c06d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f845c0709f0 tx=0x7f844c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:06.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:06.904+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f84440557c0 con 0x7f845c06c770 2023-07-04T18:52:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.132+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f845c04e210 con 0x7f845c06c770 2023-07-04T18:52:07.134 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.132+0000 7f8453fff700 1 -- 172.21.15.77:0/2071913895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7f8444058de0 con 0x7f845c06c770 2023-07-04T18:52:07.134 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:07.135 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:52:07.137 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 msgr2=0x7f843c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 0x7f843c06d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f845c0709f0 tx=0x7f844c009040 comp rx=0 tx=0).stop 2023-07-04T18:52:07.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 msgr2=0x7f845c0cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.138 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f844400bee0 tx=0x7f8444004750 comp rx=0 tx=0).stop 2023-07-04T18:52:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 shutdown_connections 2023-07-04T18:52:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f843c06ace0 0x7f843c06d190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f845c06c770 0x7f845c0cb590 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.139 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f845c159280 0x7f845c14dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 --2- 172.21.15.77:0/2071913895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f845c0cdca0 0x7f845c1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 >> 172.21.15.77:0/2071913895 conn(0x7f845c06bba0 msgr2=0x7f845c158480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 shutdown_connections 2023-07-04T18:52:07.140 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.134+0000 7f8461fcd700 1 -- 172.21.15.77:0/2071913895 wait complete. 2023-07-04T18:52:07.141 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:52:07.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:07.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.268+0000 7feecb463700 1 Processor -- start 2023-07-04T18:52:07.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.268+0000 7feecb463700 1 -- start start 2023-07-04T18:52:07.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feecb463700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7feec40ce9b0 0x7feec4164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feecb463700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feec406a040 0x7feec406a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feecb463700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec4072100 con 0x7feec4071420 2023-07-04T18:52:07.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feecb463700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec40ccdf0 con 0x7feec406a040 2023-07-04T18:52:07.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feecb463700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec40cd040 con 0x7feec40ce9b0 2023-07-04T18:52:07.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec89fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feec4071420 0x7feec4164e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52672/0 (socket says 172.21.15.77:52672) 2023-07-04T18:52:07.280 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec89fe700 1 -- 172.21.15.77:0/3476606381 learned_addr learned my addr 172.21.15.77:0/3476606381 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:07.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 --2- 172.21.15.77:0/3476606381 >> v2:172.21.15.77:3300/0 conn(0x7feec406a040 0x7feec406a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec91ff700 1 --2- 172.21.15.77:0/3476606381 >> v2:172.21.15.82:3300/0 conn(0x7feec40ce9b0 0x7feec4164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feeb7fff700 1 -- 172.21.15.77:0/3476606381 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409415178 0 0) 0x7feeb8001110 con 0x7feec4071420 2023-07-04T18:52:07.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feeb7fff700 1 -- 172.21.15.77:0/3476606381 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb00035b0 con 0x7feec4071420 2023-07-04T18:52:07.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 -- 172.21.15.77:0/3476606381 >> v2:172.21.15.82:3300/0 conn(0x7feec40ce9b0 msgr2=0x7feec4164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:07.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 --2- 172.21.15.77:0/3476606381 >> v2:172.21.15.82:3300/0 conn(0x7feec40ce9b0 0x7feec4164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 -- 172.21.15.77:0/3476606381 >> v1:172.21.15.77:6789/0 conn(0x7feec4071420 legacy=0x7feec4164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 -- 172.21.15.77:0/3476606381 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4070880 con 0x7feec406a040 2023-07-04T18:52:07.283 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.269+0000 7feec9a00700 1 --2- 172.21.15.77:0/3476606381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec406a040 0x7feec406a4a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feec4066180 tx=0x7feebc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0d9d28f15cd0f3 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feeb7fff700 1 -- 172.21.15.77:0/3476606381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00e070 con 0x7feec406a040 2023-07-04T18:52:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feeb7fff700 1 -- 172.21.15.77:0/3476606381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc00bb20 con 0x7feec406a040 2023-07-04T18:52:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feeb7fff700 1 -- 172.21.15.77:0/3476606381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc012430 con 0x7feec406a040 2023-07-04T18:52:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 -- 172.21.15.77:0/3476606381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec406a040 msgr2=0x7feec406a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 --2- 172.21.15.77:0/3476606381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec406a040 0x7feec406a4a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7feec4066180 tx=0x7feebc00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:07.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 -- 172.21.15.77:0/3476606381 shutdown_connections 2023-07-04T18:52:07.285 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 --2- 172.21.15.77:0/3476606381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec406a040 0x7feec406a4a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 --2- 172.21.15.77:0/3476606381 >> v2:172.21.15.82:3300/0 conn(0x7feec40ce9b0 0x7feec4164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 -- 172.21.15.77:0/3476606381 >> 172.21.15.77:0/3476606381 conn(0x7feec40cf220 msgr2=0x7feec40cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 -- 172.21.15.77:0/3476606381 shutdown_connections 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.270+0000 7feecb463700 1 -- 172.21.15.77:0/3476606381 wait complete. 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.271+0000 7feecb463700 1 Processor -- start 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 -- start start 2023-07-04T18:52:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 0x7feec41e5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feec4071420 0x7feec409d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec409fe30 con 0x7feec40ce9b0 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec40a0110 con 0x7feec4071420 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feecb463700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec40c62f0 con 0x7feec406a040 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feec9a00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feec9a00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45544/0 (socket says 172.21.15.77:45544) 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.272+0000 7feec9a00700 1 -- 172.21.15.77:0/2372454514 learned_addr learned my addr 172.21.15.77:0/2372454514 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec91ff700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 0x7feec41e5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec89fe700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feec4071420 0x7feec409d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 -- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 msgr2=0x7feec41e5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 0x7feec41e5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 -- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feec4071420 msgr2=0x7feec409d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feec4071420 0x7feec409d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 -- 172.21.15.77:0/2372454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc009a70 con 0x7feec40ce9b0 2023-07-04T18:52:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec91ff700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 0x7feec41e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feec9a00700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feebc014040 tx=0x7feebc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc012a70 con 0x7feec40ce9b0 2023-07-04T18:52:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc020070 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc012e30 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec40c65d0 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.273+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec41e5750 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.274+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec40ae9f0 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.274+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7feebc004040 con 0x7feec40ce9b0 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.275+0000 7feeb5ffb700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 0x7feea806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.291 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.275+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7feebc022070 con 0x7feec40ce9b0 2023-07-04T18:52:07.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.275+0000 7feec91ff700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 0x7feea806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.276+0000 7feec91ff700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 0x7feea806d190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feec0009c30 tx=0x7feec0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.292 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.279+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7feebc0bd310 con 0x7feec40ce9b0 2023-07-04T18:52:07.511 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.510+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feec409ff70 con 0x7feec40ce9b0 2023-07-04T18:52:07.512 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.510+0000 7feeb5ffb700 1 -- 172.21.15.77:0/2372454514 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4659 (secure 0 0 0) 0x7feec409ff70 con 0x7feec40ce9b0 2023-07-04T18:52:07.512 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:07.513 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/45087914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":45087914},{"type":"v1","addr":"172.21.15.82:6836","nonce":45087914}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/1097486034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1097486034},{"type":"v1","addr":"172.21.15.77:6834","nonce":1097486034}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/1661333876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":1661333876},{"type":"v1","addr":"172.21.15.77:6835","nonce":1661333876}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:51:21.845433+0000","modified":"2023-07-04T18:51:25.489990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6837/3922191311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":3922191311},{"type":"v1","addr":"172.21.15.82:6837","nonce":3922191311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-04T18:52:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 msgr2=0x7feea806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 0x7feea806d190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feec0009c30 tx=0x7feec0008040 comp rx=0 tx=0).stop 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 msgr2=0x7feec41e5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feebc014040 tx=0x7feebc004ad0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 shutdown_connections 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7feea806ace0 0x7feea806d190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.515 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feec40ce9b0 0x7feec41e5210 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7feec406a040 0x7feec41e5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 --2- 172.21.15.77:0/2372454514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feec4071420 0x7feec409d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 >> 172.21.15.77:0/2372454514 conn(0x7feec40cf220 msgr2=0x7feec41e8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.512+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 shutdown_connections 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.513+0000 7feecb463700 1 -- 172.21.15.77:0/2372454514 wait complete. 2023-07-04T18:52:07.516 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2023-07-04T18:52:07.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:07.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.646+0000 7f1948ee4700 1 Processor -- start 2023-07-04T18:52:07.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.646+0000 7f1948ee4700 1 -- start start 2023-07-04T18:52:07.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f194415c640 0x7f19440cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f194415ca10 0x7f1944070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19440cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1944071210 con 0x7f194415c640 2023-07-04T18:52:07.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19440717f0 con 0x7f194415ca10 2023-07-04T18:52:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1948ee4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19440cd290 con 0x7f1944070db0 2023-07-04T18:52:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f194415c640 0x7f19440cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f194415c640 0x7f19440cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45548/0 (socket says 172.21.15.77:45548) 2023-07-04T18:52:07.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 -- 172.21.15.77:0/2004814334 learned_addr learned my addr 172.21.15.77:0/2004814334 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:07.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 -- 172.21.15.77:0/2004814334 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 msgr2=0x7f19440cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1942d9d700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19440cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f1941d9b700 1 --2- 172.21.15.77:0/2004814334 >> v2:172.21.15.82:3300/0 conn(0x7f194415ca10 0x7f1944070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19440cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 -- 172.21.15.77:0/2004814334 >> v2:172.21.15.82:3300/0 conn(0x7f194415ca10 msgr2=0x7f1944070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 --2- 172.21.15.77:0/2004814334 >> v2:172.21.15.82:3300/0 conn(0x7f194415ca10 0x7f1944070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 -- 172.21.15.77:0/2004814334 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19440ccfb0 con 0x7f194415c640 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194259c700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19440cf120 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f19440695f0 tx=0x7f1938009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d8f9f8d0a62521 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194159a700 1 -- 172.21.15.77:0/2004814334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800b870 con 0x7f194415c640 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194159a700 1 -- 172.21.15.77:0/2004814334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193800b9d0 con 0x7f194415c640 2023-07-04T18:52:07.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.647+0000 7f194159a700 1 -- 172.21.15.77:0/2004814334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800bd60 con 0x7f194415c640 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 -- 172.21.15.77:0/2004814334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 msgr2=0x7f19440cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19440cf120 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f19440695f0 tx=0x7f1938009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 -- 172.21.15.77:0/2004814334 shutdown_connections 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19440cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 --2- 172.21.15.77:0/2004814334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19440cf120 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 --2- 172.21.15.77:0/2004814334 >> v2:172.21.15.82:3300/0 conn(0x7f194415ca10 0x7f1944070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 -- 172.21.15.77:0/2004814334 >> 172.21.15.77:0/2004814334 conn(0x7f194406bba0 msgr2=0x7f19440b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 -- 172.21.15.77:0/2004814334 shutdown_connections 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.648+0000 7f1948ee4700 1 -- 172.21.15.77:0/2004814334 wait complete. 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.649+0000 7f1948ee4700 1 Processor -- start 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.649+0000 7f1948ee4700 1 -- start start 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19441f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 0x7f19441f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1944157f90 con 0x7f194415c640 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1944154730 con 0x7f194415ca10 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1948ee4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19441f5720 con 0x7f1944070db0 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45554/0 (socket says 172.21.15.77:45554) 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 -- 172.21.15.77:0/3256605250 learned_addr learned my addr 172.21.15.77:0/3256605250 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:07.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f194259c700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19441f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1942d9d700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 0x7f19441f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 -- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 msgr2=0x7f19441f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19441f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 -- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 msgr2=0x7f19441f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 0x7f19441f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1941d9b700 1 -- 172.21.15.77:0/3256605250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938009550 con 0x7f194415c640 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.650+0000 7f1942d9d700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 0x7f19441f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:07.665 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1941d9b700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f193400c820 tx=0x7f193400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934011800 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1934011960 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19441f39d0 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934011cf0 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.651+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19441581e0 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.652+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19440cc240 con 0x7f194415c640 2023-07-04T18:52:07.666 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.652+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1934018360 con 0x7f194415c640 2023-07-04T18:52:07.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.652+0000 7f1932ffd700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 0x7f192406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.652+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7f1934089330 con 0x7f194415c640 2023-07-04T18:52:07.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.653+0000 7f194259c700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 0x7f192406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.667 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.653+0000 7f194259c700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 0x7f192406d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1938014040 tx=0x7f1938008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.668 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.657+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1934054430 con 0x7f194415c640 2023-07-04T18:52:07.847 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.846+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f194404e210 con 0x7f194415c640 2023-07-04T18:52:07.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.847+0000 7f1932ffd700 1 -- 172.21.15.77:0/3256605250 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18338 (secure 0 0 0) 0x7f1934057a50 con 0x7f194415c640 2023-07-04T18:52:07.850 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:07.850 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:04.372702+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:51:20.418518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-07-04T18:51:20.805906+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:07.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 msgr2=0x7f192406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 0x7f192406d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1938014040 tx=0x7f1938008030 comp rx=0 tx=0).stop 2023-07-04T18:52:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 msgr2=0x7f19441f3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f193400c820 tx=0x7f193400cbe0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 shutdown_connections 2023-07-04T18:52:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f192406ace0 0x7f192406d190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194415ca10 0x7f19441f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1944070db0 0x7f19441f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 --2- 172.21.15.77:0/3256605250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f194415c640 0x7f19441f3490 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 >> 172.21.15.77:0/3256605250 conn(0x7f194406bba0 msgr2=0x7f19440add50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 shutdown_connections 2023-07-04T18:52:07.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.849+0000 7f1948ee4700 1 -- 172.21.15.77:0/3256605250 wait complete. 2023-07-04T18:52:07.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-04T18:52:07.987 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 Processor -- start 2023-07-04T18:52:07.987 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 -- start start 2023-07-04T18:52:07.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf4c0cf920 0x7fdf4c06ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0cf100 con 0x7fdf4c15a430 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c15df60 con 0x7fdf4c0cf920 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf52319700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c15e240 con 0x7fdf4c15ea40 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf508b6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf4c0cf920 0x7fdf4c06ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.986+0000 7fdf508b6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf4c0cf920 0x7fdf4c06ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45564/0 (socket says 172.21.15.77:45564) 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 -- 172.21.15.77:0/2349079268 learned_addr learned my addr 172.21.15.77:0/2349079268 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:07.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf4b7fe700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 -- 172.21.15.77:0/2349079268 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 msgr2=0x7fdf4c0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 -- 172.21.15.77:0/2349079268 >> v1:172.21.15.82:6789/0 conn(0x7fdf4c15a430 legacy=0x7fdf4c0ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 -- 172.21.15.77:0/2349079268 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c15e520 con 0x7fdf4c0cf920 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf508b6700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c06ef20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009e50 tx=0x7fdf4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4ae051bc6fe3fd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:07.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf4affd700 1 -- 172.21.15.77:0/2349079268 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400e070 con 0x7fdf4c0cf920 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf4affd700 1 -- 172.21.15.77:0/2349079268 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf44003a20 con 0x7fdf4c0cf920 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.987+0000 7fdf4affd700 1 -- 172.21.15.77:0/2349079268 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf44003dd0 con 0x7fdf4c0cf920 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 -- 172.21.15.77:0/2349079268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 msgr2=0x7fdf4c06ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c06ef20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009e50 tx=0x7fdf4400b330 comp rx=0 tx=0).stop 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 -- 172.21.15.77:0/2349079268 shutdown_connections 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c06ef20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 --2- 172.21.15.77:0/2349079268 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 -- 172.21.15.77:0/2349079268 >> 172.21.15.77:0/2349079268 conn(0x7fdf4c06d7c0 msgr2=0x7fdf4c06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:07.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 -- 172.21.15.77:0/2349079268 shutdown_connections 2023-07-04T18:52:07.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.988+0000 7fdf52319700 1 -- 172.21.15.77:0/2349079268 wait complete. 2023-07-04T18:52:07.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.989+0000 7fdf52319700 1 Processor -- start 2023-07-04T18:52:07.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.989+0000 7fdf52319700 1 -- start start 2023-07-04T18:52:07.997 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf4c144b70 0x7fdf4c144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0b0610 con 0x7fdf4c0cf920 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0b08f0 con 0x7fdf4c144b70 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0b0bd0 con 0x7fdf4c15ea40 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.998 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45566/0 (socket says 172.21.15.77:45566) 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 -- 172.21.15.77:0/1294416357 learned_addr learned my addr 172.21.15.77:0/1294416357 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4b7fe700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf508b6700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf4c144b70 0x7fdf4c144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 -- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 msgr2=0x7fdf4c0b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 -- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf4c144b70 msgr2=0x7fdf4c144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:07.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf4c144b70 0x7fdf4c144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 -- 172.21.15.77:0/1294416357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44009b30 con 0x7fdf4c0cf920 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf4bfff700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400c4d0 tx=0x7fdf3400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34011790 con 0x7fdf4c0cf920 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf340118f0 con 0x7fdf4c0cf920 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c0b0eb0 con 0x7fdf4c0cf920 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.990+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34011ca0 con 0x7fdf4c0cf920 2023-07-04T18:52:08.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.991+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c0b2770 con 0x7fdf4c0cf920 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.991+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c04e210 con 0x7fdf4c0cf920 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.992+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdf3401c460 con 0x7fdf4c0cf920 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.992+0000 7fdf48ff9700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 0x7fdf2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.992+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6120+0+0 (secure 0 0 0) 0x7fdf340891e0 con 0x7fdf4c0cf920 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.992+0000 7fdf4b7fe700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 0x7fdf2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.993+0000 7fdf4b7fe700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 0x7fdf2c06d190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0b1e90 tx=0x7fdf3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:08.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:07.996+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdf34054250 con 0x7fdf4c0cf920 2023-07-04T18:52:08.224 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.223+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdf4c0ce270 con 0x7fdf4c0cf920 2023-07-04T18:52:08.225 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T18:52:08.223+0000 7f6a7a377700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-04T18:52:08.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.229+0000 7fdf48ff9700 1 -- 172.21.15.77:0/1294416357 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdf34057870 con 0x7fdf4c0cf920 2023-07-04T18:52:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 msgr2=0x7fdf2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 0x7fdf2c06d190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0b1e90 tx=0x7fdf3c008040 comp rx=0 tx=0).stop 2023-07-04T18:52:08.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 msgr2=0x7fdf4c0b16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400c4d0 tx=0x7fdf3400c7e0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 shutdown_connections 2023-07-04T18:52:08.236 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fdf2c06ace0 0x7fdf2c06d190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf4c0cf920 0x7fdf4c0b16b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf4c144b70 0x7fdf4c144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.237 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 --2- 172.21.15.77:0/1294416357 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fdf4c15ea40 0x7fdf4c0b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 >> 172.21.15.77:0/1294416357 conn(0x7fdf4c06d7c0 msgr2=0x7fdf4c157fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 shutdown_connections 2023-07-04T18:52:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.231+0000 7fdf52319700 1 -- 172.21.15.77:0/1294416357 wait complete. 2023-07-04T18:52:08.238 INFO:teuthology.orchestra.run.smithi077.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-04T18:52:08.245 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-04T18:52:08.270 INFO:tasks.ceph.mds.b.smithi082.stdout:starting mds.b at 2023-07-04T18:52:08.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.367+0000 7f34f5593700 1 Processor -- start 2023-07-04T18:52:08.369 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 -- start start 2023-07-04T18:52:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34f00cd8c0 0x7f34f006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f34f006d570 0x7f34f00ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f0158cd0 con 0x7f34f00cebd0 2023-07-04T18:52:08.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f015c750 con 0x7f34f00cd8c0 2023-07-04T18:52:08.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34f5593700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f015ca30 con 0x7f34f006d570 2023-07-04T18:52:08.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34eeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34f00cd8c0 0x7f34f006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34eeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f34f00cd8c0 0x7f34f006d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45574/0 (socket says 172.21.15.77:45574) 2023-07-04T18:52:08.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.368+0000 7f34eeffd700 1 -- 172.21.15.77:0/4021344332 learned_addr learned my addr 172.21.15.77:0/4021344332 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34ee7fc700 1 --2- 172.21.15.77:0/4021344332 >> v2:172.21.15.82:3300/0 conn(0x7f34f006d570 0x7f34f00ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34eeffd700 1 -- 172.21.15.77:0/4021344332 >> v2:172.21.15.82:3300/0 conn(0x7f34f006d570 msgr2=0x7f34f00ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34eeffd700 1 --2- 172.21.15.77:0/4021344332 >> v2:172.21.15.82:3300/0 conn(0x7f34f006d570 0x7f34f00ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34eeffd700 1 -- 172.21.15.77:0/4021344332 >> v1:172.21.15.82:6789/0 conn(0x7f34f00cebd0 legacy=0x7f34f006f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34eeffd700 1 -- 172.21.15.77:0/4021344332 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f00cf030 con 0x7f34f00cd8c0 2023-07-04T18:52:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34eeffd700 1 --2- 172.21.15.77:0/4021344332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cd8c0 0x7f34f006d030 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f34f00713b0 tx=0x7f34e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63f847f4f2ca58d server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34edffb700 1 -- 172.21.15.77:0/4021344332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4010800 con 0x7f34f00cd8c0 2023-07-04T18:52:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34edffb700 1 -- 172.21.15.77:0/4021344332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4010960 con 0x7f34f00cd8c0 2023-07-04T18:52:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.369+0000 7f34edffb700 1 -- 172.21.15.77:0/4021344332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4010cf0 con 0x7f34f00cd8c0 2023-07-04T18:52:08.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 -- 172.21.15.77:0/4021344332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cd8c0 msgr2=0x7f34f006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 --2- 172.21.15.77:0/4021344332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cd8c0 0x7f34f006d030 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f34f00713b0 tx=0x7f34e4009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 -- 172.21.15.77:0/4021344332 shutdown_connections 2023-07-04T18:52:08.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 --2- 172.21.15.77:0/4021344332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cd8c0 0x7f34f006d030 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 --2- 172.21.15.77:0/4021344332 >> v2:172.21.15.82:3300/0 conn(0x7f34f006d570 0x7f34f00ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 -- 172.21.15.77:0/4021344332 >> 172.21.15.77:0/4021344332 conn(0x7f34f0065bd0 msgr2=0x7f34f0065fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:08.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 -- 172.21.15.77:0/4021344332 shutdown_connections 2023-07-04T18:52:08.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.370+0000 7f34f5593700 1 -- 172.21.15.77:0/4021344332 wait complete. 2023-07-04T18:52:08.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.371+0000 7f34f5593700 1 Processor -- start 2023-07-04T18:52:08.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.371+0000 7f34f5593700 1 -- start start 2023-07-04T18:52:08.381 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34f006d570 0x7f34f0143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34f00cd8c0 0x7f34f00b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00ab840 con 0x7f34f00cebd0 2023-07-04T18:52:08.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f015b4b0 con 0x7f34f006d570 2023-07-04T18:52:08.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34f5593700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f015b790 con 0x7f34f00cd8c0 2023-07-04T18:52:08.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45584/0 (socket says 172.21.15.77:45584) 2023-07-04T18:52:08.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 -- 172.21.15.77:0/3469494730 learned_addr learned my addr 172.21.15.77:0/3469494730 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:08.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ee7fc700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34f00cd8c0 0x7f34f00b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34eeffd700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34f006d570 0x7f34f0143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 -- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34f00cd8c0 msgr2=0x7f34f00b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34f00cd8c0 0x7f34f00b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 -- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34f006d570 msgr2=0x7f34f0143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:08.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34f006d570 0x7f34f0143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:08.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 -- 172.21.15.77:0/3469494730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e4009550 con 0x7f34f00cebd0 2023-07-04T18:52:08.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.372+0000 7f34ef7fe700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f34e800cb50 tx=0x7f34e800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:08.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.373+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e8003c00 con 0x7f34f00cebd0 2023-07-04T18:52:08.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.373+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f00717d0 con 0x7f34f00cebd0 2023-07-04T18:52:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.373+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e8003d60 con 0x7f34f00cebd0 2023-07-04T18:52:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.373+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e800aa20 con 0x7f34f00cebd0 2023-07-04T18:52:08.389 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.373+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f0143d60 con 0x7f34f00cebd0 2023-07-04T18:52:08.390 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.374+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f00aba10 con 0x7f34f00cebd0 2023-07-04T18:52:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.374+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f34e800ab80 con 0x7f34f00cebd0 2023-07-04T18:52:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.374+0000 7f34d77fe700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 0x7f34dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.375+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f34e80895a0 con 0x7f34f00cebd0 2023-07-04T18:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.375+0000 7f34eeffd700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 0x7f34dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.375+0000 7f34eeffd700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 0x7f34dc06d190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009c10 tx=0x7f34e4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.379+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f34f00aba10 con 0x7f34f00cebd0 2023-07-04T18:52:08.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:08.608+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f34f015cd90 con 0x7f34f00cebd0 2023-07-04T18:52:09.230 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.228+0000 7f34d77fe700 1 -- 172.21.15.77:0/3469494730 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f34e8054540 con 0x7f34f00cebd0 2023-07-04T18:52:09.233 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 msgr2=0x7f34dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 0x7f34dc06d190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009c10 tx=0x7f34e4008030 comp rx=0 tx=0).stop 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 msgr2=0x7f34f00b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f34e800cb50 tx=0x7f34e800ce60 comp rx=0 tx=0).stop 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 shutdown_connections 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f34dc06ace0 0x7f34dc06d190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34f00cebd0 0x7f34f00b32d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.234 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34f006d570 0x7f34f0143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 --2- 172.21.15.77:0/3469494730 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f34f00cd8c0 0x7f34f00b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 >> 172.21.15.77:0/3469494730 conn(0x7f34f0065bd0 msgr2=0x7f34f00cb240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 shutdown_connections 2023-07-04T18:52:09.235 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.231+0000 7f34f5593700 1 -- 172.21.15.77:0/3469494730 wait complete. 2023-07-04T18:52:09.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-04T18:52:09.373 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.372+0000 7f90fac10700 1 Processor -- start 2023-07-04T18:52:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.372+0000 7f90fac10700 1 -- start start 2023-07-04T18:52:09.375 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90fac10700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f90f415a490 0x7f90f40cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90fac10700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90f40cc930 0x7f90f40705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90fac10700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f40cd8e0 con 0x7f90f4156f80 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90fac10700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f40b4f80 con 0x7f90f40cc930 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90fac10700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f40ccd30 con 0x7f90f415a490 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f90f40cc930 0x7f90f40705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f89ac700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f90f4156f80 0x7f90f406ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52726/0 (socket says 172.21.15.77:52726) 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f89ac700 1 -- 172.21.15.77:0/2904079159 learned_addr learned my addr 172.21.15.77:0/2904079159 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 -- 172.21.15.77:0/2904079159 >> v2:172.21.15.82:3300/0 conn(0x7f90f415a490 msgr2=0x7f90f40cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 --2- 172.21.15.77:0/2904079159 >> v2:172.21.15.82:3300/0 conn(0x7f90f415a490 0x7f90f40cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 -- 172.21.15.77:0/2904079159 >> v1:172.21.15.77:6789/0 conn(0x7f90f4156f80 legacy=0x7f90f406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 -- 172.21.15.77:0/2904079159 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f40b2a90 con 0x7f90f40cc930 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f37fe700 1 -- 172.21.15.77:0/2904079159 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120929640 0 0) 0x7f90f40cd8e0 con 0x7f90f4156f80 2023-07-04T18:52:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.373+0000 7f90f91ad700 1 --2- 172.21.15.77:0/2904079159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f40705e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f90f406fb90 tx=0x7f90ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f283f2a79f53a5d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90f37fe700 1 -- 172.21.15.77:0/2904079159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00e070 con 0x7f90f40cc930 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90f37fe700 1 -- 172.21.15.77:0/2904079159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec00bbd0 con 0x7f90f40cc930 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90f37fe700 1 -- 172.21.15.77:0/2904079159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec012430 con 0x7f90f40cc930 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 -- 172.21.15.77:0/2904079159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 msgr2=0x7f90f40705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 --2- 172.21.15.77:0/2904079159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f40705e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f90f406fb90 tx=0x7f90ec00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 -- 172.21.15.77:0/2904079159 shutdown_connections 2023-07-04T18:52:09.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 --2- 172.21.15.77:0/2904079159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f40705e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.379 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 --2- 172.21.15.77:0/2904079159 >> v2:172.21.15.82:3300/0 conn(0x7f90f415a490 0x7f90f40cc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 -- 172.21.15.77:0/2904079159 >> 172.21.15.77:0/2904079159 conn(0x7f90f406bbe0 msgr2=0x7f90f40b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:09.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 -- 172.21.15.77:0/2904079159 shutdown_connections 2023-07-04T18:52:09.380 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.374+0000 7f90fac10700 1 -- 172.21.15.77:0/2904079159 wait complete. 2023-07-04T18:52:09.382 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.375+0000 7f90fac10700 1 Processor -- start 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.375+0000 7f90fac10700 1 -- start start 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f4156f80 0x7f90f41554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90f415a490 0x7f90f415be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f41effe0 con 0x7f90f40cc930 2023-07-04T18:52:09.383 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f41f02c0 con 0x7f90f4156f80 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90fac10700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f41f05a0 con 0x7f90f415a490 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90f89ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90f89ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45594/0 (socket says 172.21.15.77:45594) 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90f89ac700 1 -- 172.21.15.77:0/605582434 learned_addr learned my addr 172.21.15.77:0/605582434 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.376+0000 7f90f3fff700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f4156f80 0x7f90f41554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:09.384 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f91ad700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90f415a490 0x7f90f415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 -- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90f415a490 msgr2=0x7f90f415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90f415a490 0x7f90f415be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 -- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f4156f80 msgr2=0x7f90f41554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f4156f80 0x7f90f41554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:09.385 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 -- 172.21.15.77:0/605582434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec009a70 con 0x7f90f40cc930 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f89ac700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f90e80011e0 tx=0x7f90e800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8004c80 con 0x7f90f40cc930 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8004e00 con 0x7f90f40cc930 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f41f0880 con 0x7f90f40cc930 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8007310 con 0x7f90f40cc930 2023-07-04T18:52:09.386 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.377+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f41f2e20 con 0x7f90f40cc930 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.378+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f404e210 con 0x7f90f40cc930 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.378+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90e8007470 con 0x7f90f40cc930 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.379+0000 7f90f17fa700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 0x7f90d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.379+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6206+0+0 (secure 0 0 0) 0x7f90e80892b0 con 0x7f90f40cc930 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.379+0000 7f90f3fff700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 0x7f90d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:09.387 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.379+0000 7f90f3fff700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 0x7f90d806d190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006fd0 tx=0x7f90e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:09.388 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.383+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90e80542d0 con 0x7f90f40cc930 2023-07-04T18:52:09.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:09.573+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f90f4070fb0 con 0x7f90f40cc930 2023-07-04T18:52:10.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.411+0000 7f90f17fa700 1 -- 172.21.15.77:0/605582434 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v27) v1 ==== 181+0+0 (secure 0 0 0) 0x7f90e80578f0 con 0x7f90f40cc930 2023-07-04T18:52:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 msgr2=0x7f90d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 0x7f90d806d190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006fd0 tx=0x7f90e0008040 comp rx=0 tx=0).stop 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 msgr2=0x7f90f41f28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f90e80011e0 tx=0x7f90e800cae0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 shutdown_connections 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.415+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f90d806ace0 0x7f90d806d190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f90f40cc930 0x7f90f41f28e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f90f415a490 0x7f90f415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 --2- 172.21.15.77:0/605582434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f4156f80 0x7f90f41554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 >> 172.21.15.77:0/605582434 conn(0x7f90f406bbe0 msgr2=0x7f90f406c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 shutdown_connections 2023-07-04T18:52:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.416+0000 7f90fac10700 1 -- 172.21.15.77:0/605582434 wait complete. 2023-07-04T18:52:10.419 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' removed 2023-07-04T18:52:10.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-04T18:52:10.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.566+0000 7f6c20c1e700 1 Processor -- start 2023-07-04T18:52:10.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 -- start start 2023-07-04T18:52:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f6c1c15a490 0x7f6c1c0cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c1c0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0cd8e0 con 0x7f6c1c156f80 2023-07-04T18:52:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0b4f80 con 0x7f6c1c15a490 2023-07-04T18:52:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.567+0000 7f6c20c1e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0ccd50 con 0x7f6c1c0cc930 2023-07-04T18:52:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1a59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6c1c156f80 0x7f6c1c06ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52740/0 (socket says 172.21.15.77:52740) 2023-07-04T18:52:10.572 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1a59c700 1 -- 172.21.15.77:0/3736065049 learned_addr learned my addr 172.21.15.77:0/3736065049 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/3736065049 >> v2:172.21.15.82:3300/0 conn(0x7f6c1c15a490 0x7f6c1c0cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c1c0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021778622 0 0) 0x7f6c1c0cd8e0 con 0x7f6c1c156f80 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c040035b0 con 0x7f6c1c156f80 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955201135 0 0) 0x7f6c040035b0 con 0x7f6c1c156f80 2023-07-04T18:52:10.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 msgr2=0x7f6c1c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c1c0705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 >> v2:172.21.15.82:3300/0 conn(0x7f6c1c15a490 msgr2=0x7f6c1c0cc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 --2- 172.21.15.77:0/3736065049 >> v2:172.21.15.82:3300/0 conn(0x7f6c1c15a490 0x7f6c1c0cc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0b2a90 con 0x7f6c1c156f80 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f6c10001670 con 0x7f6c1c156f80 2023-07-04T18:52:10.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 >> v1:172.21.15.77:6789/0 conn(0x7f6c1c156f80 legacy=0x7f6c1c06ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c04003a00 0x7f6c04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c04005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0b4f80 con 0x7f6c04004d90 2023-07-04T18:52:10.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0ccd50 con 0x7f6c04003a00 2023-07-04T18:52:10.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0400ba10 con 0x7f6c1c0cc930 2023-07-04T18:52:10.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c04005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c1a59c700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c04003a00 0x7f6c04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.568+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 -- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 msgr2=0x7f6c04004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c0cc930 0x7f6c04004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 -- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c04003a00 msgr2=0x7f6c04003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c04003a00 0x7f6c04003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 -- 172.21.15.77:0/3736065049 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040035b0 con 0x7f6c04004d90 2023-07-04T18:52:10.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c04005200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009fc0 tx=0x7f6c04005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e355d0dba0b26d46 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400ebd0 con 0x7f6c04004d90 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400f040 con 0x7f6c04004d90 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1959a700 1 -- 172.21.15.77:0/3736065049 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14014480 con 0x7f6c04004d90 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c20c1e700 1 -- 172.21.15.77:0/3736065049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 msgr2=0x7f6c04005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/3736065049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c04005200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009fc0 tx=0x7f6c04005880 comp rx=0 tx=0).stop 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c1ad9d700 1 -- 172.21.15.77:0/3736065049 reap_dead start 2023-07-04T18:52:10.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c20c1e700 1 -- 172.21.15.77:0/3736065049 shutdown_connections 2023-07-04T18:52:10.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.569+0000 7f6c20c1e700 1 -- 172.21.15.77:0/3736065049 >> 172.21.15.77:0/3736065049 conn(0x7f6c1c06bbe0 msgr2=0x7f6c1c0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:10.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.570+0000 7f6c20c1e700 1 -- 172.21.15.77:0/3736065049 shutdown_connections 2023-07-04T18:52:10.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.570+0000 7f6c20c1e700 1 -- 172.21.15.77:0/3736065049 wait complete. 2023-07-04T18:52:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.571+0000 7f6c20c1e700 1 Processor -- start 2023-07-04T18:52:10.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.571+0000 7f6c20c1e700 1 -- start start 2023-07-04T18:52:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 0x7f6c1c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 0x7f6c1c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c1506a0 con 0x7f6c04004d90 2023-07-04T18:52:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14fa90 con 0x7f6c1c14f0d0 2023-07-04T18:52:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14fd70 con 0x7f6c1c151070 2023-07-04T18:52:10.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45614/0 (socket says 172.21.15.77:45614) 2023-07-04T18:52:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 -- 172.21.15.77:0/51504681 learned_addr learned my addr 172.21.15.77:0/51504681 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 0x7f6c1c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 0x7f6c1c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 -- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 msgr2=0x7f6c1c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 0x7f6c1c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 -- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 msgr2=0x7f6c1c150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 0x7f6c1c150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 -- 172.21.15.77:0/51504681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c150050 con 0x7f6c04004d90 2023-07-04T18:52:10.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1ad9d700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 0x7f6c1c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 0x7f6c1c150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c1a59c700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6c10007310 tx=0x7f6c100076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c100077f0 con 0x7f6c04004d90 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1000f8d0 con 0x7f6c04004d90 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c150330 con 0x7f6c04004d90 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000fc80 con 0x7f6c04004d90 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.572+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c14e870 con 0x7f6c04004d90 2023-07-04T18:52:10.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.573+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c150980 con 0x7f6c04004d90 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.574+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6c1001d460 con 0x7f6c04004d90 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.574+0000 7f6c0affd700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 0x7f6bfc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.574+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5768+0+0 (secure 0 0 0) 0x7f6c100891e0 con 0x7f6c04004d90 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.574+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 0x7f6bfc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.575+0000 7f6c19d9b700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 0x7f6bfc06d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009bd0 tx=0x7f6c0c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:10.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.578+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6c10054440 con 0x7f6c04004d90 2023-07-04T18:52:10.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:10.766+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c1c14e870 con 0x7f6c04004d90 2023-07-04T18:52:11.419 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.417+0000 7f6c0affd700 1 -- 172.21.15.77:0/51504681 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v28) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6c1c14e870 con 0x7f6c04004d90 2023-07-04T18:52:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 msgr2=0x7f6bfc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 0x7f6bfc06d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009bd0 tx=0x7f6c0c008040 comp rx=0 tx=0).stop 2023-07-04T18:52:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 msgr2=0x7f6c1c14e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6c10007310 tx=0x7f6c100076d0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 shutdown_connections 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f6bfc06ace0 0x7f6bfc06d190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c04004d90 0x7f6c1c14e150 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f6c1c151070 0x7f6c1c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 --2- 172.21.15.77:0/51504681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c1c14f0d0 0x7f6c1c150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 >> 172.21.15.77:0/51504681 conn(0x7f6c1c06bbe0 msgr2=0x7f6c1c15d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 shutdown_connections 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.419+0000 7f6c20c1e700 1 -- 172.21.15.77:0/51504681 wait complete. 2023-07-04T18:52:11.423 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' removed 2023-07-04T18:52:11.436 INFO:tasks.ceph.mds.a:Restarting daemon 2023-07-04T18:52:11.436 INFO:tasks.ceph.mds.a:Stopping old one... 2023-07-04T18:52:11.437 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-07-04T18:52:11.437 INFO:teuthology.orchestra.run:waiting for 300 2023-07-04T18:52:11.438 INFO:tasks.ceph.mds.c:Restarting daemon 2023-07-04T18:52:11.438 INFO:tasks.ceph.mds.c:Stopping old one... 2023-07-04T18:52:11.439 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-07-04T18:52:11.439 INFO:teuthology.orchestra.run:waiting for 300 2023-07-04T18:52:11.439 INFO:tasks.ceph.mds.b:Restarting daemon 2023-07-04T18:52:11.439 INFO:tasks.ceph.mds.b:Stopping old one... 2023-07-04T18:52:11.440 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-07-04T18:52:11.440 INFO:teuthology.orchestra.run:waiting for 300 2023-07-04T18:52:11.440 INFO:tasks.ceph.mds.d:Restarting daemon 2023-07-04T18:52:11.440 INFO:tasks.ceph.mds.d:Stopping old one... 2023-07-04T18:52:11.441 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-07-04T18:52:11.441 INFO:teuthology.orchestra.run:waiting for 300 2023-07-04T18:52:11.447 INFO:tasks.ceph.mds.a:Stopped 2023-07-04T18:52:11.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-07-04T18:52:11.448 INFO:tasks.ceph.mds.c:Stopped 2023-07-04T18:52:11.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-07-04T18:52:11.450 INFO:tasks.ceph.mds.d:Stopped 2023-07-04T18:52:11.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-07-04T18:52:11.451 INFO:tasks.ceph.mds.b:Stopped 2023-07-04T18:52:11.452 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-07-04T18:52:11.490 INFO:tasks.ceph.mds.a:Started 2023-07-04T18:52:11.491 DEBUG:teuthology.parallel:result is None 2023-07-04T18:52:11.492 INFO:tasks.ceph.mds.c:Started 2023-07-04T18:52:11.492 DEBUG:teuthology.parallel:result is None 2023-07-04T18:52:11.493 INFO:tasks.ceph.mds.d:Started 2023-07-04T18:52:11.493 DEBUG:teuthology.parallel:result is None 2023-07-04T18:52:11.494 INFO:tasks.ceph.mds.b:Started 2023-07-04T18:52:11.494 DEBUG:teuthology.parallel:result is None 2023-07-04T18:52:11.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-07-04T18:52:11.573 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2023-07-04T18:52:11.574 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2023-07-04T18:52:11.575 INFO:tasks.ceph.mds.b.smithi082.stdout:starting mds.b at 2023-07-04T18:52:11.575 INFO:tasks.ceph.mds.d.smithi082.stdout:starting mds.d at 2023-07-04T18:52:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.623+0000 7f3415752700 1 Processor -- start 2023-07-04T18:52:11.625 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.623+0000 7f3415752700 1 -- start start 2023-07-04T18:52:11.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f3415752700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f341015c510 0x7f3410160030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f3415752700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f3415752700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34100b0270 con 0x7f34100cc580 2023-07-04T18:52:11.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f3415752700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341015c8e0 con 0x7f341015c510 2023-07-04T18:52:11.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f3415752700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3410160f90 con 0x7f341006ef20 2023-07-04T18:52:11.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340f7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f34100cc580 0x7f34100cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52788/0 (socket says 172.21.15.77:52788) 2023-07-04T18:52:11.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340f7fe700 1 -- 172.21.15.77:0/2453507937 learned_addr learned my addr 172.21.15.77:0/2453507937 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:11.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340effd700 1 --2- 172.21.15.77:0/2453507937 >> v2:172.21.15.82:3300/0 conn(0x7f341015c510 0x7f3410160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340e7fc700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691511732 0 0) 0x7f34100b0270 con 0x7f34100cc580 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.624+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f80035b0 con 0x7f34100cc580 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340effd700 1 -- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 msgr2=0x7f34100cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340effd700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214909698 0 0) 0x7f33f80035b0 con 0x7f34100cc580 2023-07-04T18:52:11.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340effd700 1 -- 172.21.15.77:0/2453507937 >> v1:172.21.15.77:6789/0 conn(0x7f34100cc580 legacy=0x7f34100cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340effd700 1 -- 172.21.15.77:0/2453507937 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100ce7e0 con 0x7f341015c510 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340e7fc700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340effd700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f3410160030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3404009840 tx=0x7f3404009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31dad973e689edf8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34080019f0 con 0x7f341015c510 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3404004030 con 0x7f341015c510 2023-07-04T18:52:11.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.625+0000 7f340dffb700 1 -- 172.21.15.77:0/2453507937 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34040043a0 con 0x7f341015c510 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 -- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 msgr2=0x7f3410160030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f3410160030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3404009840 tx=0x7f3404009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 -- 172.21.15.77:0/2453507937 shutdown_connections 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f3410160030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 --2- 172.21.15.77:0/2453507937 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 -- 172.21.15.77:0/2453507937 >> 172.21.15.77:0/2453507937 conn(0x7f341006d7c0 msgr2=0x7f341006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 -- 172.21.15.77:0/2453507937 shutdown_connections 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.626+0000 7f3415752700 1 -- 172.21.15.77:0/2453507937 wait complete. 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.627+0000 7f3415752700 1 Processor -- start 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.627+0000 7f3415752700 1 -- start start 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f34100c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100b39a0 con 0x7f34100cc580 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100c6c80 con 0x7f341015c510 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34100ad860 con 0x7f341006ef20 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45654/0 (socket says 172.21.15.77:45654) 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 -- 172.21.15.77:0/2100540349 learned_addr learned my addr 172.21.15.77:0/2100540349 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:11.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340effd700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340f7fe700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f34100c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 -- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 msgr2=0x7f34100ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 -- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 msgr2=0x7f34100c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f34100c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 -- 172.21.15.77:0/2100540349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34000094a0 con 0x7f34100cc580 2023-07-04T18:52:11.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340f7fe700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f34100c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f340e7fc700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f340000ebe0 tx=0x7f340000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340000c850 con 0x7f34100cc580 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340000c9b0 con 0x7f34100cc580 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040094a0 con 0x7f34100cc580 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340000cd40 con 0x7f34100cc580 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.628+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100ade20 con 0x7f34100cc580 2023-07-04T18:52:11.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.629+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34100ae750 con 0x7f34100cc580 2023-07-04T18:52:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.630+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3400013470 con 0x7f34100cc580 2023-07-04T18:52:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.630+0000 7f33f77fe700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 0x7f33fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.630+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5296+0+0 (secure 0 0 0) 0x7f34000891f0 con 0x7f34100cc580 2023-07-04T18:52:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.630+0000 7f340effd700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 0x7f33fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:11.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.631+0000 7f340effd700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 0x7f33fc06d190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3404000c00 tx=0x7f3404008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:11.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.635+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f34000545b0 con 0x7f34100cc580 2023-07-04T18:52:11.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:11.838+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f341004e210 con 0x7f34100cc580 2023-07-04T18:52:13.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.334+0000 7f33f77fe700 1 -- 172.21.15.77:0/2100540349 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f340000cea0 con 0x7f34100cc580 2023-07-04T18:52:13.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 msgr2=0x7f33fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.339 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 0x7f33fc06d190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3404000c00 tx=0x7f3404008030 comp rx=0 tx=0).stop 2023-07-04T18:52:13.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 msgr2=0x7f341014dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f340000ebe0 tx=0x7f340000eef0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 shutdown_connections 2023-07-04T18:52:13.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f33fc06ace0 0x7f33fc06d190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f341015c510 0x7f34100c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f341006ef20 0x7f34100ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 --2- 172.21.15.77:0/2100540349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34100cc580 0x7f341014dfc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 >> 172.21.15.77:0/2100540349 conn(0x7f341006d7c0 msgr2=0x7f34100cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:13.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 shutdown_connections 2023-07-04T18:52:13.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.337+0000 7f3415752700 1 -- 172.21.15.77:0/2100540349 wait complete. 2023-07-04T18:52:13.342 INFO:teuthology.orchestra.run.smithi077.stderr: removed all blocklist entries 2023-07-04T18:52:13.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-07-04T18:52:13.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.482+0000 7fd55094a700 1 Processor -- start 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.482+0000 7fd55094a700 1 -- start start 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd55094a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd54c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd55094a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c06b750 con 0x7fd54c15d950 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd55094a700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0701c0 con 0x7fd54c0cb750 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd55094a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c072150 con 0x7fd54c06b2b0 2023-07-04T18:52:13.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd549d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd54c15d950 0x7fd54c0c9c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52804/0 (socket says 172.21.15.77:52804) 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd549d9b700 1 -- 172.21.15.77:0/2852553684 learned_addr learned my addr 172.21.15.77:0/2852553684 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54ad9d700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd54c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427290485 0 0) 0x7fd54c06b750 con 0x7fd54c15d950 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5380035b0 con 0x7fd54c15d950 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376516641 0 0) 0x7fd5380035b0 con 0x7fd54c15d950 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 msgr2=0x7fd54c0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:13.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd54c0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 >> v1:172.21.15.82:6789/0 conn(0x7fd54c0cb750 legacy=0x7fd54c15d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c071b90 con 0x7fd54c15d950 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fd534001670 con 0x7fd54c15d950 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 >> v1:172.21.15.77:6789/0 conn(0x7fd54c15d950 legacy=0x7fd54c0c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd538003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd538003f70 0x7fd538004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd538004d90 0x7fd538005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c072150 con 0x7fd538003f70 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53800ba40 con 0x7fd538004d90 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53800de10 con 0x7fd54c06b2b0 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd549d9b700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd538003f70 0x7fd538004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54a59c700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd538003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.483+0000 7fd54ad9d700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd538004d90 0x7fd538005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 -- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 msgr2=0x7fd538003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd538003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 -- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd538004d90 msgr2=0x7fd538005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd538004d90 0x7fd538005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 -- 172.21.15.77:0/2852553684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5380035b0 con 0x7fd538003f70 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd54a59c700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd54c06b2b0 0x7fd538003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd549d9b700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd538003f70 0x7fd538004850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd54c15dd40 tx=0x7fd538005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ffc442e95dbd01 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53400ce00 con 0x7fd538003f70 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534005960 con 0x7fd538003f70 2023-07-04T18:52:13.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd54959a700 1 -- 172.21.15.77:0/2852553684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534005d10 con 0x7fd538003f70 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd55094a700 1 -- 172.21.15.77:0/2852553684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd538003f70 msgr2=0x7fd538004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd55094a700 1 --2- 172.21.15.77:0/2852553684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd538003f70 0x7fd538004850 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd54c15dd40 tx=0x7fd538005880 comp rx=0 tx=0).stop 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd54ad9d700 1 -- 172.21.15.77:0/2852553684 reap_dead start 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd55094a700 1 -- 172.21.15.77:0/2852553684 shutdown_connections 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd55094a700 1 -- 172.21.15.77:0/2852553684 >> 172.21.15.77:0/2852553684 conn(0x7fd54c0cf220 msgr2=0x7fd54c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.484+0000 7fd55094a700 1 -- 172.21.15.77:0/2852553684 shutdown_connections 2023-07-04T18:52:13.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.485+0000 7fd55094a700 1 -- 172.21.15.77:0/2852553684 wait complete. 2023-07-04T18:52:13.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.486+0000 7fd55094a700 1 Processor -- start 2023-07-04T18:52:13.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.486+0000 7fd55094a700 1 -- start start 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 0x7fd54c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 0x7fd54c1f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c155bb0 con 0x7fd54c0c5010 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0c3760 con 0x7fd54c09ff90 2023-07-04T18:52:13.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c159030 con 0x7fd538003f70 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45680/0 (socket says 172.21.15.77:45680) 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 -- 172.21.15.77:0/929015226 learned_addr learned my addr 172.21.15.77:0/929015226 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd549d9b700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 0x7fd54c1f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54a59c700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 0x7fd54c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 -- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 msgr2=0x7fd54c0c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 0x7fd54c0c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 -- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 msgr2=0x7fd54c1f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 0x7fd54c1f1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 -- 172.21.15.77:0/929015226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c1592a0 con 0x7fd54c0c5010 2023-07-04T18:52:13.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54a59c700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 0x7fd54c0c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:13.497 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd549d9b700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 0x7fd54c1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd54ad9d700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd5440107f0 tx=0x7fd544010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544004300 con 0x7fd54c0c5010 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd544004460 con 0x7fd54c0c5010 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c1f2130 con 0x7fd54c0c5010 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440038e0 con 0x7fd54c0c5010 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.487+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0c5580 con 0x7fd54c0c5010 2023-07-04T18:52:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.488+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c159b80 con 0x7fd54c0c5010 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.489+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd544003a40 con 0x7fd54c0c5010 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.489+0000 7fd542ffd700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 0x7fd52c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.489+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd544088eb0 con 0x7fd54c0c5010 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.489+0000 7fd54a59c700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 0x7fd52c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.490+0000 7fd54a59c700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 0x7fd52c06d190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001520 tx=0x7fd53c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.493+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd5440542b0 con 0x7fd54c0c5010 2023-07-04T18:52:13.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.755+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd54c04e210 con 0x7fd54c0c5010 2023-07-04T18:52:13.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.755+0000 7fd542ffd700 1 -- 172.21.15.77:0/929015226 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd5440578d0 con 0x7fd54c0c5010 2023-07-04T18:52:13.756 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:13.756 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-07-04T18:52:13.756 INFO:teuthology.orchestra.run.smithi077.stdout: "auth_dump": [ 2023-07-04T18:52:13.756 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.757 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.a", 2023-07-04T18:52:13.757 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRky+nFKxAAezOJXbPY3CsDRiEcGgACTA==", 2023-07-04T18:52:13.757 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.757 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.758 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.b", 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRky/VMOhAAXB5gnz520M0KrkGtvr1MAw==", 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.759 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.c", 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRkAH0oNRAAAZUQHvxEfVyCKzH6njiD5g==", 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.760 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mds.d", 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwaaRkaJZLBhAA3CaTJFrHF9QmApJPgMDtvw==", 2023-07-04T18:52:13.761 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.0", 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDzaaRk61edIRAAdAERV/JhvfaF//RGvGuOHg==", 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.1", 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD2aaRk6fnhIBAAf8KlJRpZqp7VVp5MFYmX0Q==", 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.764 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.2", 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD5aaRkqKUWIBAAjKLo1hLm/xz62mRUBZB8KQ==", 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.3", 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQD8aaRkSkuPHxAApozzYyTZeymH06LWVdfDGA==", 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.766 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.767 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.4", 2023-07-04T18:52:13.768 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQACaqRk7l7KIhAAdYdujaxh0QN9VlQ8xzb/ZA==", 2023-07-04T18:52:13.768 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.768 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.768 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.768 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.5", 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAFaqRku2JfIhAATRwR47BbceGZqHKEdF5Tmw==", 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.6", 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQAIaqRkLmVMIRAATAKAgx2ikSIIA/gJdz7Pvw==", 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "osd.7", 2023-07-04T18:52:13.771 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQALaqRkmJlRIBAAiBPJBA2xrcidHRyPI7zYvA==", 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow profile osd", 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile osd", 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.772 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.0", 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwaaRkpsg9CxAA3I+2BiR3flfH2NGjwz9V9Q==", 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.773 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.1", 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwaaRk+q1aFBAAkptDNcBtzvvWqbPBOunKlA==", 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2023-07-04T18:52:13.774 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.2", 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwaaRkZ1v8EBAAzkcz24OS5W6SAApn3aR7JA==", 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.775 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.3", 2023-07-04T18:52:13.776 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDwaaRknsfHFxAAMNKVa7AqjR4vqoBVuVDQVw==", 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow", 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow r", 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow rw", 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow rwx" 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.777 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.admin", 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A==", 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": "allow *", 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow *", 2023-07-04T18:52:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mds", 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkm9S1IBAAl7C5b05ZGxQRz+xhGJ9Kbw==", 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.779 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mds" 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-mgr", 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkAN+1IBAA+/otteuqmJ3gdEc/jatoEg==", 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.780 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-mgr" 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-osd", 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkWea1IBAAsPeevfQf8+DgSRKzRT0xMA==", 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-osd" 2023-07-04T18:52:13.781 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd", 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkE+21IBAA3Dc90e5R31o+YEgyEiJ5KQ==", 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd" 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.782 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkTPS1IBAA+HS7ZOTd6snBDbxqrLtbaw==", 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.783 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "client.bootstrap-rgw", 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQARaqRkhPu1IBAAIF94CYrWh80YIRnWHrNV7A==", 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile bootstrap-rgw" 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.784 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.x", 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRkY0JoKBAAsA6b/+K5BpY9FyOrOjJgFQ==", 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "entity": "mgr.y", 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "AQDvaaRkI/tdJBAAcEwV7OEFu5QIKthn3bPbuw==", 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "caps": { 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": "allow *", 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": "allow profile mgr", 2023-07-04T18:52:13.786 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": "allow *" 2023-07-04T18:52:13.787 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.787 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:13.787 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2023-07-04T18:52:13.787 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-07-04T18:52:13.788 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 msgr2=0x7fd52c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.788 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 0x7fd52c06d190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001520 tx=0x7fd53c009040 comp rx=0 tx=0).stop 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 msgr2=0x7fd54c1f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd5440107f0 tx=0x7fd544010bb0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 shutdown_connections 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd52c06ace0 0x7fd52c06d190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd54c0c5010 0x7fd54c1f1860 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd538003f70 0x7fd54c0c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.789 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 --2- 172.21.15.77:0/929015226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd54c09ff90 0x7fd54c1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 >> 172.21.15.77:0/929015226 conn(0x7fd54c0cf220 msgr2=0x7fd54c0ca680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:13.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 shutdown_connections 2023-07-04T18:52:13.790 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.757+0000 7fd55094a700 1 -- 172.21.15.77:0/929015226 wait complete. 2023-07-04T18:52:13.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-07-04T18:52:13.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.911+0000 7ff15337f700 1 Processor -- start 2023-07-04T18:52:13.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.911+0000 7ff15337f700 1 -- start start 2023-07-04T18:52:13.915 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15337f700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff14c15ca10 0x7ff14c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.915 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15337f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.915 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15337f700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0cf830 con 0x7ff14c15c640 2023-07-04T18:52:13.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15337f700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0717f0 con 0x7ff14c15ca10 2023-07-04T18:52:13.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15337f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0cd290 con 0x7ff14c070db0 2023-07-04T18:52:13.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15091a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff14c15ca10 0x7ff14c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.916 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15191c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15091a700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff14c15ca10 0x7ff14c070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:51492/0 (socket says 172.21.15.77:51492) 2023-07-04T18:52:13.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.912+0000 7ff15191c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:35538/0 (socket says 172.21.15.77:35538) 2023-07-04T18:52:13.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 -- 172.21.15.77:0/1049177189 learned_addr learned my addr 172.21.15.77:0/1049177189 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:13.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 -- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 msgr2=0x7ff14c0cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 -- 172.21.15.77:0/1049177189 >> v1:172.21.15.82:6789/0 conn(0x7ff14c15c640 legacy=0x7ff14c0cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 -- 172.21.15.77:0/1049177189 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c071210 con 0x7ff14c15ca10 2023-07-04T18:52:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15191c700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff15091a700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c070870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1480098f0 tx=0x7ff148009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f780755847382cb server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff143fff700 1 -- 172.21.15.77:0/1049177189 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148011830 con 0x7ff14c15ca10 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff143fff700 1 -- 172.21.15.77:0/1049177189 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff148011990 con 0x7ff14c15ca10 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.913+0000 7ff143fff700 1 -- 172.21.15.77:0/1049177189 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148011d20 con 0x7ff14c15ca10 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 -- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 msgr2=0x7ff14c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c070870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1480098f0 tx=0x7ff148009c00 comp rx=0 tx=0).stop 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 -- 172.21.15.77:0/1049177189 shutdown_connections 2023-07-04T18:52:13.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c070db0 0x7ff14c0cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 --2- 172.21.15.77:0/1049177189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c070870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 -- 172.21.15.77:0/1049177189 >> 172.21.15.77:0/1049177189 conn(0x7ff14c06bba0 msgr2=0x7ff14c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:13.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 -- 172.21.15.77:0/1049177189 shutdown_connections 2023-07-04T18:52:13.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.914+0000 7ff15337f700 1 -- 172.21.15.77:0/1049177189 wait complete. 2023-07-04T18:52:13.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.915+0000 7ff15337f700 1 Processor -- start 2023-07-04T18:52:13.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.915+0000 7ff15337f700 1 -- start start 2023-07-04T18:52:13.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c15c640 0x7ff14c1f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c8190 con 0x7ff14c070db0 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0c8d10 con 0x7ff14c15ca10 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15337f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c0b0d30 con 0x7ff14c15c640 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45690/0 (socket says 172.21.15.77:45690) 2023-07-04T18:52:13.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 -- 172.21.15.77:0/3055469002 learned_addr learned my addr 172.21.15.77:0/3055469002 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15091a700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c15c640 0x7ff14c1f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15191c700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 -- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c15c640 msgr2=0x7ff14c1f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c15c640 0x7ff14c1f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 -- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 msgr2=0x7ff14c0b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c0b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:13.924 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 -- 172.21.15.77:0/3055469002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff144009a70 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff15111b700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c4e0 tx=0x7ff13c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.916+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0052d0 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.917+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c005430 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.917+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff148009550 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.917+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0038c0 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.917+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c0c86c0 con 0x7ff14c070db0 2023-07-04T18:52:13.925 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.918+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c04e210 con 0x7ff14c070db0 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.918+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff13c003a20 con 0x7ff14c070db0 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.918+0000 7ff141ffb700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 0x7ff13006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.918+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff13c0884a0 con 0x7ff14c070db0 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.919+0000 7ff15091a700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 0x7ff13006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.919+0000 7ff15091a700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 0x7ff13006d190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff148009c40 tx=0x7ff14801d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:13.926 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:13.923+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff13c0538b0 con 0x7ff14c070db0 2023-07-04T18:52:14.188 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.187+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7ff14c15c0d0 con 0x7ff14c070db0 2023-07-04T18:52:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.190+0000 7ff141ffb700 1 -- 172.21.15.77:0/3055469002 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff13c056ed0 con 0x7ff14c070db0 2023-07-04T18:52:14.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 msgr2=0x7ff13006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 0x7ff13006d190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff148009c40 tx=0x7ff14801d040 comp rx=0 tx=0).stop 2023-07-04T18:52:14.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 msgr2=0x7ff14c1598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00c4e0 tx=0x7ff13c00c7f0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.193 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 shutdown_connections 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff13006ace0 0x7ff13006d190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff14c070db0 0x7ff14c1598e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff14c15ca10 0x7ff14c0b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 --2- 172.21.15.77:0/3055469002 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff14c15c640 0x7ff14c1f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 >> 172.21.15.77:0/3055469002 conn(0x7ff14c06bba0 msgr2=0x7ff14c15a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 shutdown_connections 2023-07-04T18:52:14.194 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.192+0000 7ff15337f700 1 -- 172.21.15.77:0/3055469002 wait complete. 2023-07-04T18:52:14.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-07-04T18:52:14.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.327+0000 7f13afe26700 1 Processor -- start 2023-07-04T18:52:14.328 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.328+0000 7f13afe26700 1 -- start start 2023-07-04T18:52:14.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.328+0000 7f13afe26700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f13a815d950 0x7f13a806b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.328+0000 7f13afe26700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a80701c0 con 0x7f13a80c9c50 2023-07-04T18:52:14.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.328+0000 7f13afe26700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a8071b90 con 0x7f13a80cb750 2023-07-04T18:52:14.331 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.328+0000 7f13afe26700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a8072150 con 0x7f13a815d950 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ae3c3700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f13a80c9c50 0x7f13a80ca0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52840/0 (socket says 172.21.15.77:52840) 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ae3c3700 1 -- 172.21.15.77:0/3222651171 learned_addr learned my addr 172.21.15.77:0/3222651171 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922651473 0 0) 0x7f13a80701c0 con 0x7f13a80c9c50 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/3222651171 >> v2:172.21.15.82:3300/0 conn(0x7f13a815d950 0x7f13a806b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13940035b0 con 0x7f13a80c9c50 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628142523 0 0) 0x7f13940035b0 con 0x7f13a80c9c50 2023-07-04T18:52:14.332 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 >> v2:172.21.15.82:3300/0 conn(0x7f13a815d950 msgr2=0x7f13a806b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 --2- 172.21.15.77:0/3222651171 >> v2:172.21.15.82:3300/0 conn(0x7f13a815d950 0x7f13a806b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 >> v1:172.21.15.82:6789/0 conn(0x7f13a80cb750 legacy=0x7f13a815d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8071e70 con 0x7f13a80c9c50 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f13a00019f0 con 0x7f13a80c9c50 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 >> v1:172.21.15.77:6789/0 conn(0x7f13a80c9c50 legacy=0x7f13a80ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1394003a00 0x7f1394003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1394004850 0x7f1394004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f139400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8072150 con 0x7f1394005200 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139400c080 con 0x7f1394003a00 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139400c240 con 0x7f1394004850 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f139400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13adbc2700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1394003a00 0x7f1394003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1394004850 0x7f1394004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.334 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ae3c3700 1 -- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1394004850 msgr2=0x7f1394004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.329+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f1394004850 0x7f1394004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13ae3c3700 1 -- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1394003a00 msgr2=0x7f1394003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1394003a00 0x7f1394003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13ae3c3700 1 -- 172.21.15.77:0/3222651171 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940035b0 con 0x7f1394005200 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13ae3c3700 1 -- 172.21.15.77:0/3222651171 reap_dead start 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f139400ba10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f13a000a830 tx=0x7f1394005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e76181fbca7d5b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a000a2f0 con 0x7f1394005200 2023-07-04T18:52:14.335 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0012620 con 0x7f1394005200 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13acbc0700 1 -- 172.21.15.77:0/3222651171 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a00129d0 con 0x7f1394005200 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 -- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 msgr2=0x7f139400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f139400ba10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f13a000a830 tx=0x7f1394005880 comp rx=0 tx=0).stop 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 -- 172.21.15.77:0/3222651171 shutdown_connections 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 --2- 172.21.15.77:0/3222651171 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f139400ba10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 -- 172.21.15.77:0/3222651171 >> 172.21.15.77:0/3222651171 conn(0x7f13a80cf220 msgr2=0x7f13a80cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:14.336 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 -- 172.21.15.77:0/3222651171 shutdown_connections 2023-07-04T18:52:14.337 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.330+0000 7f13afe26700 1 -- 172.21.15.77:0/3222651171 wait complete. 2023-07-04T18:52:14.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 Processor -- start 2023-07-04T18:52:14.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 -- start start 2023-07-04T18:52:14.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 0x7f13a81ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.340 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 0x7f13a81eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a806e1b0 con 0x7f1394005200 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80a3400 con 0x7f13a81ee750 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13afe26700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a81ef110 con 0x7f13a80adfc0 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13adbc2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13adbc2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45710/0 (socket says 172.21.15.77:45710) 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.332+0000 7f13adbc2700 1 -- 172.21.15.77:0/158908594 learned_addr learned my addr 172.21.15.77:0/158908594 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:14.341 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 0x7f13a81ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 0x7f13a81eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 -- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 msgr2=0x7f13a81ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 0x7f13a81ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 -- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 msgr2=0x7f13a81eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 0x7f13a81eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 -- 172.21.15.77:0/158908594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a806d5a0 con 0x7f1394005200 2023-07-04T18:52:14.342 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 0x7f13a81ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13ae3c3700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 0x7f13a81eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13adbc2700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f139800f850 tx=0x7f139800fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980050d0 con 0x7f1394005200 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1398005230 con 0x7f1394005200 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398003680 con 0x7f1394005200 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a806d880 con 0x7f1394005200 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.333+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a806de40 con 0x7f1394005200 2023-07-04T18:52:14.343 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.334+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a804e210 con 0x7f1394005200 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.334+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f13980037e0 con 0x7f1394005200 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.335+0000 7f139e7fc700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 0x7f138c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.335+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1398088e30 con 0x7f1394005200 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.335+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 0x7f138c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.335+0000 7f13ad3c1700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 0x7f138c06d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13a80aeb30 tx=0x7f13a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.344 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.339+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f13980542c0 con 0x7f1394005200 2023-07-04T18:52:14.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.603+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f13a8159b90 con 0x7f1394005200 2023-07-04T18:52:14.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.607+0000 7f139e7fc700 1 -- 172.21.15.77:0/158908594 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f13980578e0 con 0x7f1394005200 2023-07-04T18:52:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 msgr2=0x7f138c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 0x7f138c06d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13a80aeb30 tx=0x7f13a4008040 comp rx=0 tx=0).stop 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 msgr2=0x7f13a80a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f139800f850 tx=0x7f139800fc10 comp rx=0 tx=0).stop 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 shutdown_connections 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f138c06ace0 0x7f138c06d190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1394005200 0x7f13a80a2ec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13a81ee750 0x7f13a81eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 --2- 172.21.15.77:0/158908594 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13a80adfc0 0x7f13a81ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 >> 172.21.15.77:0/158908594 conn(0x7f13a80cf220 msgr2=0x7f13a80cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.609+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 shutdown_connections 2023-07-04T18:52:14.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.610+0000 7f13afe26700 1 -- 172.21.15.77:0/158908594 wait complete. 2023-07-04T18:52:14.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-07-04T18:52:14.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.742+0000 7f441f6d0700 1 Processor -- start 2023-07-04T18:52:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.742+0000 7f441f6d0700 1 -- start start 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441f6d0700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f441815a430 0x7f44180ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441f6d0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f44180cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441f6d0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44180cf100 con 0x7f441815ea40 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441f6d0700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441815df60 con 0x7f441815a430 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441f6d0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441815e240 con 0x7f44180cebf0 2023-07-04T18:52:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441dc6d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f441815ea40 0x7f44180cf920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52860/0 (socket says 172.21.15.77:52860) 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441dc6d700 1 -- 172.21.15.77:0/3262826540 learned_addr learned my addr 172.21.15.77:0/3262826540 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679144747 0 0) 0x7f44180cf100 con 0x7f441815ea40 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441d46c700 1 --2- 172.21.15.77:0/3262826540 >> v2:172.21.15.82:3300/0 conn(0x7f441815a430 0x7f44180ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44040035b0 con 0x7f441815ea40 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f441cc6b700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f44180cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136450608 0 0) 0x7f44040035b0 con 0x7f441815ea40 2023-07-04T18:52:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 msgr2=0x7f44180cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f44180cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 >> v2:172.21.15.82:3300/0 conn(0x7f441815a430 msgr2=0x7f44180ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 --2- 172.21.15.77:0/3262826540 >> v2:172.21.15.82:3300/0 conn(0x7f441815a430 0x7f44180ce6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441815e520 con 0x7f441815ea40 2023-07-04T18:52:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f44100019f0 con 0x7f441815ea40 2023-07-04T18:52:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 >> v1:172.21.15.77:6789/0 conn(0x7f441815ea40 legacy=0x7f44180cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.743+0000 7f440ffff700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f4404003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4404003f80 0x7f4404004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4404004d90 0x7f4404005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441815df60 con 0x7f4404004d90 2023-07-04T18:52:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441815e240 con 0x7f4404003f80 2023-07-04T18:52:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440400ba10 con 0x7f44180cebf0 2023-07-04T18:52:14.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4404004d90 0x7f4404005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441d46c700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f4404003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441cc6b700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4404003f80 0x7f4404004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 -- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 msgr2=0x7f4404003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f44180cebf0 0x7f4404003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 -- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4404003f80 msgr2=0x7f4404004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4404003f80 0x7f4404004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 -- 172.21.15.77:0/3262826540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44040035b0 con 0x7f4404004d90 2023-07-04T18:52:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f441dc6d700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4404004d90 0x7f4404005200 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f441000a8d0 tx=0x7f4404005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0428ace6060e69 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441000a340 con 0x7f4404004d90 2023-07-04T18:52:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410012600 con 0x7f4404004d90 2023-07-04T18:52:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.744+0000 7f440ffff700 1 -- 172.21.15.77:0/3262826540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44100129b0 con 0x7f4404004d90 2023-07-04T18:52:14.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 -- 172.21.15.77:0/3262826540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4404004d90 msgr2=0x7f4404005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 --2- 172.21.15.77:0/3262826540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4404004d90 0x7f4404005200 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f441000a8d0 tx=0x7f4404005880 comp rx=0 tx=0).stop 2023-07-04T18:52:14.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441dc6d700 1 -- 172.21.15.77:0/3262826540 reap_dead start 2023-07-04T18:52:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 -- 172.21.15.77:0/3262826540 shutdown_connections 2023-07-04T18:52:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 -- 172.21.15.77:0/3262826540 >> 172.21.15.77:0/3262826540 conn(0x7f441806d7c0 msgr2=0x7f441806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 -- 172.21.15.77:0/3262826540 shutdown_connections 2023-07-04T18:52:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.745+0000 7f441f6d0700 1 -- 172.21.15.77:0/3262826540 wait complete. 2023-07-04T18:52:14.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.746+0000 7f441f6d0700 1 Processor -- start 2023-07-04T18:52:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.746+0000 7f441f6d0700 1 -- start start 2023-07-04T18:52:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 0x7f44180cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441814c290 0x7f44180cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44180ac8b0 con 0x7f441814bec0 2023-07-04T18:52:14.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44180c6a80 con 0x7f441814c290 2023-07-04T18:52:14.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441f6d0700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44180c6d60 con 0x7f4418154130 2023-07-04T18:52:14.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.747+0000 7f441d46c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45730/0 (socket says 172.21.15.77:45730) 2023-07-04T18:52:14.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 -- 172.21.15.77:0/2101018319 learned_addr learned my addr 172.21.15.77:0/2101018319 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441cc6b700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 0x7f44180cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441dc6d700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441814c290 0x7f44180cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 -- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 msgr2=0x7f44180cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 0x7f44180cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 -- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441814c290 msgr2=0x7f44180cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441814c290 0x7f44180cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 -- 172.21.15.77:0/2101018319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180c7040 con 0x7f441814bec0 2023-07-04T18:52:14.767 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441cc6b700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 0x7f44180cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441d46c700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4414011aa0 tx=0x7f4414011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441400f9e0 con 0x7f441814bec0 2023-07-04T18:52:14.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441400fb40 con 0x7f441814bec0 2023-07-04T18:52:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44180c7320 con 0x7f441814bec0 2023-07-04T18:52:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414014440 con 0x7f441814bec0 2023-07-04T18:52:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.748+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441806af50 con 0x7f441814bec0 2023-07-04T18:52:14.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.749+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44180acb90 con 0x7f441814bec0 2023-07-04T18:52:14.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.750+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f44140145a0 con 0x7f441814bec0 2023-07-04T18:52:14.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.750+0000 7f440dffb700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 0x7f43fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:14.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.750+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f44140899c0 con 0x7f441814bec0 2023-07-04T18:52:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.750+0000 7f441cc6b700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 0x7f43fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.751+0000 7f441cc6b700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 0x7f43fc06d190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4418154500 tx=0x7f4408008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:14.771 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:14.755+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4414054db0 con 0x7f441814bec0 2023-07-04T18:52:15.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.014+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f441804e210 con 0x7f441814bec0 2023-07-04T18:52:15.019 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.018+0000 7f440dffb700 1 -- 172.21.15.77:0/2101018319 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f44140583d0 con 0x7f441814bec0 2023-07-04T18:52:15.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 msgr2=0x7f43fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 0x7f43fc06d190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4418154500 tx=0x7f4408008040 comp rx=0 tx=0).stop 2023-07-04T18:52:15.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 msgr2=0x7f44180aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4414011aa0 tx=0x7f4414011e60 comp rx=0 tx=0).stop 2023-07-04T18:52:15.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 shutdown_connections 2023-07-04T18:52:15.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f43fc06ace0 0x7f43fc06d190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f441814bec0 0x7f44180aec30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441814c290 0x7f44180cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 --2- 172.21.15.77:0/2101018319 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4418154130 0x7f44180cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 >> 172.21.15.77:0/2101018319 conn(0x7f441806d7c0 msgr2=0x7f441815d850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 shutdown_connections 2023-07-04T18:52:15.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.020+0000 7f441f6d0700 1 -- 172.21.15.77:0/2101018319 wait complete. 2023-07-04T18:52:15.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-07-04T18:52:15.162 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 Processor -- start 2023-07-04T18:52:15.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 -- start start 2023-07-04T18:52:15.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f941415ea40 0x7f94140cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f94140cf920 0x7f941406ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140cf100 con 0x7f941415a430 2023-07-04T18:52:15.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941415df60 con 0x7f941415ea40 2023-07-04T18:52:15.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.154+0000 7f941a22e700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941415e240 con 0x7f94140cf920 2023-07-04T18:52:15.164 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f941415ea40 0x7f94140cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f941415ea40 0x7f94140cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45740/0 (socket says 172.21.15.77:45740) 2023-07-04T18:52:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 -- 172.21.15.77:0/1279750197 learned_addr learned my addr 172.21.15.77:0/1279750197 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9413fff700 1 --2- 172.21.15.77:0/1279750197 >> v2:172.21.15.82:3300/0 conn(0x7f94140cf920 0x7f941406ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 -- 172.21.15.77:0/1279750197 >> v2:172.21.15.82:3300/0 conn(0x7f94140cf920 msgr2=0x7f941406ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.172 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 --2- 172.21.15.77:0/1279750197 >> v2:172.21.15.82:3300/0 conn(0x7f94140cf920 0x7f941406ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 -- 172.21.15.77:0/1279750197 >> v1:172.21.15.82:6789/0 conn(0x7f941415a430 legacy=0x7f94140ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 -- 172.21.15.77:0/1279750197 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941415e520 con 0x7f941415ea40 2023-07-04T18:52:15.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f9412ffd700 1 --2- 172.21.15.77:0/1279750197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f941415ea40 0x7f94140cf3e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f94140c7b40 tx=0x7f9408009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7255f5b70997696c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.173 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f94127fc700 1 -- 172.21.15.77:0/1279750197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408004230 con 0x7f941415ea40 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f94127fc700 1 -- 172.21.15.77:0/1279750197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408004390 con 0x7f941415ea40 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.155+0000 7f94127fc700 1 -- 172.21.15.77:0/1279750197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94080036a0 con 0x7f941415ea40 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 -- 172.21.15.77:0/1279750197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f941415ea40 msgr2=0x7f94140cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 --2- 172.21.15.77:0/1279750197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f941415ea40 0x7f94140cf3e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f94140c7b40 tx=0x7f9408009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 -- 172.21.15.77:0/1279750197 shutdown_connections 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 --2- 172.21.15.77:0/1279750197 >> v2:172.21.15.82:3300/0 conn(0x7f94140cf920 0x7f941406ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.174 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 --2- 172.21.15.77:0/1279750197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f941415ea40 0x7f94140cf3e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 -- 172.21.15.77:0/1279750197 >> 172.21.15.77:0/1279750197 conn(0x7f941406d7c0 msgr2=0x7f941406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 -- 172.21.15.77:0/1279750197 shutdown_connections 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.156+0000 7f941a22e700 1 -- 172.21.15.77:0/1279750197 wait complete. 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.157+0000 7f941a22e700 1 Processor -- start 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.157+0000 7f941a22e700 1 -- start start 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.175 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941415ea40 0x7f94140b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94141594a0 0x7f9414144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94141e4840 con 0x7f94140cf920 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94141e3f30 con 0x7f941415ea40 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140b4080 con 0x7f94141594a0 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45744/0 (socket says 172.21.15.77:45744) 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 -- 172.21.15.77:0/2957705666 learned_addr learned my addr 172.21.15.77:0/2957705666 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:15.176 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f9412ffd700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941415ea40 0x7f94140b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 -- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94141594a0 msgr2=0x7f9414144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94141594a0 0x7f9414144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 -- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941415ea40 msgr2=0x7f94140b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941415ea40 0x7f94140b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 -- 172.21.15.77:0/2957705666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408009550 con 0x7f94140cf920 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f94137fe700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f940400c4a0 tx=0x7f940400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.177 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040054e0 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9404003680 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94140b66b0 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404003a30 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.158+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141e4460 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.159+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941404e210 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.160+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9404003b90 con 0x7f94140cf920 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.160+0000 7f93fbfff700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 0x7f940006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.178 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.160+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f94040886a0 con 0x7f94140cf920 2023-07-04T18:52:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.160+0000 7f9412ffd700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 0x7f940006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.161+0000 7f9412ffd700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 0x7f940006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f94140b2680 tx=0x7f9408008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.179 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.165+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9404053aa0 con 0x7f94140cf920 2023-07-04T18:52:15.430 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.429+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f94140ce270 con 0x7f94140cf920 2023-07-04T18:52:15.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.434+0000 7f93fbfff700 1 -- 172.21.15.77:0/2957705666 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f94040570c0 con 0x7f94140cf920 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 msgr2=0x7f940006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 0x7f940006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f94140b2680 tx=0x7f9408008030 comp rx=0 tx=0).stop 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 msgr2=0x7f94140b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f940400c4a0 tx=0x7f940400c860 comp rx=0 tx=0).stop 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 shutdown_connections 2023-07-04T18:52:15.438 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f940006ace0 0x7f940006d190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94140cf920 0x7f94140b6020 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f94141594a0 0x7f9414144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 --2- 172.21.15.77:0/2957705666 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941415ea40 0x7f94140b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 >> 172.21.15.77:0/2957705666 conn(0x7f941406d7c0 msgr2=0x7f94140af960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 shutdown_connections 2023-07-04T18:52:15.439 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.436+0000 7f941a22e700 1 -- 172.21.15.77:0/2957705666 wait complete. 2023-07-04T18:52:15.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-07-04T18:52:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.571+0000 7fbe6ef07700 1 Processor -- start 2023-07-04T18:52:15.573 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.571+0000 7fbe6ef07700 1 -- start start 2023-07-04T18:52:15.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6ef07700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fbe6815ca10 0x7fbe68070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6ef07700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe68070db0 0x7fbe680cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6ef07700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680cf830 con 0x7fbe6815c640 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6ef07700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe680717f0 con 0x7fbe6815ca10 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6ef07700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680cd290 con 0x7fbe68070db0 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6cca3700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbe6815c640 0x7fbe680cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52890/0 (socket says 172.21.15.77:52890) 2023-07-04T18:52:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.572+0000 7fbe6cca3700 1 -- 172.21.15.77:0/2879745983 learned_addr learned my addr 172.21.15.77:0/2879745983 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367022803 0 0) 0x7fbe680cf830 con 0x7fbe6815c640 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe68070db0 0x7fbe680cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe580035b0 con 0x7fbe6815c640 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe67fff700 1 --2- 172.21.15.77:0/2879745983 >> v2:172.21.15.82:3300/0 conn(0x7fbe6815ca10 0x7fbe68070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755940691 0 0) 0x7fbe580035b0 con 0x7fbe6815c640 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe68070db0 msgr2=0x7fbe680cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.578 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe68070db0 0x7fbe680cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 >> v2:172.21.15.82:3300/0 conn(0x7fbe6815ca10 msgr2=0x7fbe68070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 --2- 172.21.15.77:0/2879745983 >> v2:172.21.15.82:3300/0 conn(0x7fbe6815ca10 0x7fbe68070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68071210 con 0x7fbe6815c640 2023-07-04T18:52:15.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fbe5c001670 con 0x7fbe6815c640 2023-07-04T18:52:15.579 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 >> v1:172.21.15.77:6789/0 conn(0x7fbe6815c640 legacy=0x7fbe680cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe58003a00 0x7fbe58003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe58004850 0x7fbe58004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe58005200 0x7fbe5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680717f0 con 0x7fbe58005200 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680cd290 con 0x7fbe58003a00 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5800c130 con 0x7fbe58004850 2023-07-04T18:52:15.580 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe67fff700 1 --2- 172.21.15.77:0/2879745983 >> v2:172.21.15.82:3300/0 conn(0x7fbe6815ca10 0x7fbe68070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe68070db0 0x7fbe680cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe58005200 0x7fbe5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe67fff700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe58004850 0x7fbe58004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6cca3700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe58003a00 0x7fbe58003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 -- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe58004850 msgr2=0x7fbe58004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe58004850 0x7fbe58004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.581 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 -- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe58003a00 msgr2=0x7fbe58003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe58003a00 0x7fbe58003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.573+0000 7fbe6d4a4700 1 -- 172.21.15.77:0/2879745983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe580035b0 con 0x7fbe58005200 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe58005200 0x7fbe5800ba10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbe68071180 tx=0x7fbe58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642a486f974badb8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60007dc0 con 0x7fbe58005200 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6000f040 con 0x7fbe58005200 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe677fe700 1 -- 172.21.15.77:0/2879745983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60003c60 con 0x7fbe58005200 2023-07-04T18:52:15.582 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6ef07700 1 -- 172.21.15.77:0/2879745983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe58005200 msgr2=0x7fbe5800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/2879745983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe58005200 0x7fbe5800ba10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbe68071180 tx=0x7fbe58005880 comp rx=0 tx=0).stop 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6d4a4700 1 -- 172.21.15.77:0/2879745983 reap_dead start 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6ef07700 1 -- 172.21.15.77:0/2879745983 shutdown_connections 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.574+0000 7fbe6ef07700 1 -- 172.21.15.77:0/2879745983 >> 172.21.15.77:0/2879745983 conn(0x7fbe6806bba0 msgr2=0x7fbe680b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.575+0000 7fbe6ef07700 1 -- 172.21.15.77:0/2879745983 shutdown_connections 2023-07-04T18:52:15.583 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.575+0000 7fbe6ef07700 1 -- 172.21.15.77:0/2879745983 wait complete. 2023-07-04T18:52:15.585 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.576+0000 7fbe6ef07700 1 Processor -- start 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.576+0000 7fbe6ef07700 1 -- start start 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe68158ef0 0x7fbe680c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe681f5460 0x7fbe681f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6815b0a0 con 0x7fbe6815a6f0 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c9db0 con 0x7fbe68158ef0 2023-07-04T18:52:15.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680b2bc0 con 0x7fbe681f5460 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45766/0 (socket says 172.21.15.77:45766) 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 -- 172.21.15.77:0/1947377817 learned_addr learned my addr 172.21.15.77:0/1947377817 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6cca3700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe68158ef0 0x7fbe680c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6d4a4700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe681f5460 0x7fbe681f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 -- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe681f5460 msgr2=0x7fbe681f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe681f5460 0x7fbe681f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 -- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe68158ef0 msgr2=0x7fbe680c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe68158ef0 0x7fbe680c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 -- 172.21.15.77:0/1947377817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe680b2ea0 con 0x7fbe6815a6f0 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe67fff700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011900 tx=0x7fbe54011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54004230 con 0x7fbe6815a6f0 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe54004390 con 0x7fbe6815a6f0 2023-07-04T18:52:15.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe680b3180 con 0x7fbe6815a6f0 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe540036a0 con 0x7fbe6815a6f0 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.577+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe680b3740 con 0x7fbe6815a6f0 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.578+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe680b2d00 con 0x7fbe6815a6f0 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.579+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbe54003800 con 0x7fbe6815a6f0 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.579+0000 7fbe657fa700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 0x7fbe4c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.579+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbe54089630 con 0x7fbe6815a6f0 2023-07-04T18:52:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.579+0000 7fbe6cca3700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 0x7fbe4c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.580+0000 7fbe6cca3700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 0x7fbe4c06d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0048d0 tx=0x7fbe5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.583+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbe680b2d00 con 0x7fbe6815a6f0 2023-07-04T18:52:15.849 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.845+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fbe6815cf20 con 0x7fbe6815a6f0 2023-07-04T18:52:15.850 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.849+0000 7fbe657fa700 1 -- 172.21.15.77:0/1947377817 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbe54089070 con 0x7fbe6815a6f0 2023-07-04T18:52:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 msgr2=0x7fbe4c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 0x7fbe4c06d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0048d0 tx=0x7fbe5c009040 comp rx=0 tx=0).stop 2023-07-04T18:52:15.853 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 msgr2=0x7fbe6815ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbe54011900 tx=0x7fbe54011cc0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 shutdown_connections 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbe4c06ace0 0x7fbe4c06d190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbe681f5460 0x7fbe681f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe68158ef0 0x7fbe680c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 --2- 172.21.15.77:0/1947377817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbe6815a6f0 0x7fbe6815ab60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 >> 172.21.15.77:0/1947377817 conn(0x7fbe6806bba0 msgr2=0x7fbe680ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.854 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 shutdown_connections 2023-07-04T18:52:15.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.851+0000 7fbe6ef07700 1 -- 172.21.15.77:0/1947377817 wait complete. 2023-07-04T18:52:15.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-07-04T18:52:15.986 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.984+0000 7ff20e71c700 1 Processor -- start 2023-07-04T18:52:15.987 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.984+0000 7ff20e71c700 1 -- start start 2023-07-04T18:52:15.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.985+0000 7ff20e71c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.989 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.985+0000 7ff20e71c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.985+0000 7ff20e71c700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080cf100 con 0x7ff20815ea40 2023-07-04T18:52:15.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.985+0000 7ff20e71c700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20815df60 con 0x7ff2080cebf0 2023-07-04T18:52:15.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.985+0000 7ff20e71c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20815e240 con 0x7ff20815a430 2023-07-04T18:52:15.990 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff2077fe700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff20ccb9700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7ff20815ea40 0x7ff2080cf920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.77:60826/0 (socket says 172.21.15.77:60826) 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3301/0 says I am v2:172.21.15.77:35622/0 (socket says 172.21.15.77:35622) 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff20ccb9700 1 -- 172.21.15.77:0/4096601486 learned_addr learned my addr 172.21.15.77:0/4096601486 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff2077fe700 1 --2- 172.21.15.77:0/4096601486 >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.77:51580/0 (socket says 172.21.15.77:51580) 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 -- 172.21.15.77:0/4096601486 >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 msgr2=0x7ff2080cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 --2- 172.21.15.77:0/4096601486 >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.991 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 -- 172.21.15.77:0/4096601486 >> v1:172.21.15.82:6789/0 conn(0x7ff20815ea40 legacy=0x7ff2080cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 -- 172.21.15.77:0/4096601486 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20815e520 con 0x7ff20815a430 2023-07-04T18:52:15.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff2077fe700 1 --2- 172.21.15.77:0/4096601486 >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:15.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff207fff700 1 --2- 172.21.15.77:0/4096601486 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009840 tx=0x7ff1fc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fc021111e75f631 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:15.992 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff206ffd700 1 -- 172.21.15.77:0/4096601486 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011830 con 0x7ff20815a430 2023-07-04T18:52:15.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff206ffd700 1 -- 172.21.15.77:0/4096601486 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc011990 con 0x7ff20815a430 2023-07-04T18:52:15.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.986+0000 7ff206ffd700 1 -- 172.21.15.77:0/4096601486 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011d20 con 0x7ff20815a430 2023-07-04T18:52:15.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 -- 172.21.15.77:0/4096601486 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 msgr2=0x7ff2080ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:15.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 --2- 172.21.15.77:0/4096601486 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009840 tx=0x7ff1fc009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:15.993 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 -- 172.21.15.77:0/4096601486 shutdown_connections 2023-07-04T18:52:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 --2- 172.21.15.77:0/4096601486 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff20815a430 0x7ff2080ce6b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 --2- 172.21.15.77:0/4096601486 >> v2:172.21.15.82:3300/0 conn(0x7ff2080cebf0 0x7ff2080cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 -- 172.21.15.77:0/4096601486 >> 172.21.15.77:0/4096601486 conn(0x7ff20806d7c0 msgr2=0x7ff20806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 -- 172.21.15.77:0/4096601486 shutdown_connections 2023-07-04T18:52:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.987+0000 7ff20e71c700 1 -- 172.21.15.77:0/4096601486 wait complete. 2023-07-04T18:52:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.988+0000 7ff20e71c700 1 Processor -- start 2023-07-04T18:52:15.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.988+0000 7ff20e71c700 1 -- start start 2023-07-04T18:52:15.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 0x7ff208151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:15.999 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff20815ea40 0x7ff2080b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b1f50 con 0x7ff2080b1ad0 2023-07-04T18:52:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b6850 con 0x7ff20815ea40 2023-07-04T18:52:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20e71c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c45f0 con 0x7ff2080cebf0 2023-07-04T18:52:16.000 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20ccb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20ccb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45782/0 (socket says 172.21.15.77:45782) 2023-07-04T18:52:16.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20ccb9700 1 -- 172.21.15.77:0/954437609 learned_addr learned my addr 172.21.15.77:0/954437609 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:16.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff207fff700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 0x7ff208151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.001 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff2077fe700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff20815ea40 0x7ff2080b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20ccb9700 1 -- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 msgr2=0x7ff208151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.989+0000 7ff20ccb9700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 0x7ff208151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20ccb9700 1 -- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff20815ea40 msgr2=0x7ff2080b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20ccb9700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff20815ea40 0x7ff2080b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20ccb9700 1 -- 172.21.15.77:0/954437609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094a0 con 0x7ff2080b1ad0 2023-07-04T18:52:16.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff207fff700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 0x7ff208151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:16.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20ccb9700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff20000ca80 tx=0x7ff20000cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000a9e0 con 0x7ff2080b1ad0 2023-07-04T18:52:16.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000ab40 con 0x7ff2080b1ad0 2023-07-04T18:52:16.003 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc0094a0 con 0x7ff2080b1ad0 2023-07-04T18:52:16.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200012400 con 0x7ff2080b1ad0 2023-07-04T18:52:16.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.990+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080b5fb0 con 0x7ff2080b1ad0 2023-07-04T18:52:16.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.991+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208067080 con 0x7ff2080b1ad0 2023-07-04T18:52:16.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.991+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff20001b3b0 con 0x7ff2080b1ad0 2023-07-04T18:52:16.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.992+0000 7ff204ff9700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 0x7ff1f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.992+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff200088c20 con 0x7ff2080b1ad0 2023-07-04T18:52:16.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.992+0000 7ff207fff700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 0x7ff1f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.992+0000 7ff207fff700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 0x7ff1f406d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc000c00 tx=0x7ff1fc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:15.996+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff200054050 con 0x7ff2080b1ad0 2023-07-04T18:52:16.261 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.260+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff20804e210 con 0x7ff2080b1ad0 2023-07-04T18:52:16.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.263+0000 7ff204ff9700 1 -- 172.21.15.77:0/954437609 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff200057670 con 0x7ff2080b1ad0 2023-07-04T18:52:16.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 msgr2=0x7ff1f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 0x7ff1f406d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc000c00 tx=0x7ff1fc008030 comp rx=0 tx=0).stop 2023-07-04T18:52:16.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 msgr2=0x7ff2080b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff20000ca80 tx=0x7ff20000cd90 comp rx=0 tx=0).stop 2023-07-04T18:52:16.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 shutdown_connections 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff1f406ace0 0x7ff1f406d190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2080b1ad0 0x7ff2080b3b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff2080cebf0 0x7ff208151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 --2- 172.21.15.77:0/954437609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff20815ea40 0x7ff2080b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.265+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 >> 172.21.15.77:0/954437609 conn(0x7ff20806d7c0 msgr2=0x7ff2080cd170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.266+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 shutdown_connections 2023-07-04T18:52:16.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.266+0000 7ff20e71c700 1 -- 172.21.15.77:0/954437609 wait complete. 2023-07-04T18:52:16.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:16.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.398+0000 7f39b4ba6700 1 Processor -- start 2023-07-04T18:52:16.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.398+0000 7f39b4ba6700 1 -- start start 2023-07-04T18:52:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f39b015a430 0x7f39b00ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b00cebf0 0x7f39b00cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b015ea40 0x7f39b00cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00cfe60 con 0x7f39b00cebf0 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00cf100 con 0x7f39b015a430 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39b4ba6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b015df60 con 0x7f39b015ea40 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39add9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b00cebf0 0x7f39b00cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39add9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f39b00cebf0 0x7f39b00cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45786/0 (socket says 172.21.15.77:45786) 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.399+0000 7f39add9b700 1 -- 172.21.15.77:0/1280170142 learned_addr learned my addr 172.21.15.77:0/1280170142 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39ae59c700 1 --2- 172.21.15.77:0/1280170142 >> v2:172.21.15.82:3300/0 conn(0x7f39b015a430 0x7f39b00ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 -- 172.21.15.77:0/1280170142 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b015ea40 msgr2=0x7f39b00cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 --2- 172.21.15.77:0/1280170142 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b015ea40 0x7f39b00cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 -- 172.21.15.77:0/1280170142 >> v2:172.21.15.82:3300/0 conn(0x7f39b015a430 msgr2=0x7f39b00ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 --2- 172.21.15.77:0/1280170142 >> v2:172.21.15.82:3300/0 conn(0x7f39b015a430 0x7f39b00ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 -- 172.21.15.77:0/1280170142 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b015e240 con 0x7f39b00cebf0 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39add9b700 1 --2- 172.21.15.77:0/1280170142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b00cebf0 0x7f39b00cf3e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f39b00d0a40 tx=0x7f39a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fc003e30fa90b4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39ad59a700 1 -- 172.21.15.77:0/1280170142 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000b870 con 0x7f39b00cebf0 2023-07-04T18:52:16.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39ad59a700 1 -- 172.21.15.77:0/1280170142 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000b9d0 con 0x7f39b00cebf0 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39ad59a700 1 -- 172.21.15.77:0/1280170142 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000bd60 con 0x7f39b00cebf0 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39b4ba6700 1 -- 172.21.15.77:0/1280170142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b00cebf0 msgr2=0x7f39b00cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/1280170142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b00cebf0 0x7f39b00cf3e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f39b00d0a40 tx=0x7f39a0009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39b4ba6700 1 -- 172.21.15.77:0/1280170142 shutdown_connections 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.400+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/1280170142 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b015ea40 0x7f39b00cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.401+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/1280170142 >> v2:172.21.15.82:3300/0 conn(0x7f39b015a430 0x7f39b00ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.401+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/1280170142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b00cebf0 0x7f39b00cf3e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.401+0000 7f39b4ba6700 1 -- 172.21.15.77:0/1280170142 >> 172.21.15.77:0/1280170142 conn(0x7f39b006d7c0 msgr2=0x7f39b006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.401+0000 7f39b4ba6700 1 -- 172.21.15.77:0/1280170142 shutdown_connections 2023-07-04T18:52:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.401+0000 7f39b4ba6700 1 -- 172.21.15.77:0/1280170142 wait complete. 2023-07-04T18:52:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.402+0000 7f39b4ba6700 1 Processor -- start 2023-07-04T18:52:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.402+0000 7f39b4ba6700 1 -- start start 2023-07-04T18:52:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39b00cebf0 0x7f39b01e3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 0x7f39b0156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b01e44a0 con 0x7f39b015ea40 2023-07-04T18:52:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00b3a40 con 0x7f39b00cebf0 2023-07-04T18:52:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00b3ce0 con 0x7f39b00c8880 2023-07-04T18:52:16.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45794/0 (socket says 172.21.15.77:45794) 2023-07-04T18:52:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 -- 172.21.15.77:0/348896554 learned_addr learned my addr 172.21.15.77:0/348896554 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39ae59c700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39b00cebf0 0x7f39b01e3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39aed9d700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 0x7f39b0156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 -- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 msgr2=0x7f39b0156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 0x7f39b0156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 -- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39b00cebf0 msgr2=0x7f39b01e3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39b00cebf0 0x7f39b01e3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 -- 172.21.15.77:0/348896554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a0009550 con 0x7f39b015ea40 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39aed9d700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 0x7f39b0156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39add9b700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f39b0071420 tx=0x7f39a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0004310 con 0x7f39b015ea40 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a001f070 con 0x7f39b015ea40 2023-07-04T18:52:16.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b00b3fc0 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00036a0 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.403+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b00b4580 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.404+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b004e210 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.405+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f39a0021030 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.405+0000 7f399effd700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 0x7f399006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.405+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f39a008a530 con 0x7f39b015ea40 2023-07-04T18:52:16.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.406+0000 7f39ae59c700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 0x7f399006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.406+0000 7f39ae59c700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 0x7f399006d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f39a400ab60 tx=0x7f39a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.410+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f39a00559c0 con 0x7f39b015ea40 2023-07-04T18:52:16.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.598+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b00ce270 con 0x7f39b015ea40 2023-07-04T18:52:16.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.599+0000 7f399effd700 1 -- 172.21.15.77:0/348896554 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+14603 (secure 0 0 0) 0x7f39a0058fe0 con 0x7f39b015ea40 2023-07-04T18:52:16.602 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:16.602 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:13.328430+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:16.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 msgr2=0x7f399006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 0x7f399006d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f39a400ab60 tx=0x7f39a4008040 comp rx=0 tx=0).stop 2023-07-04T18:52:16.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 msgr2=0x7f39b00c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f39b0071420 tx=0x7f39a0004750 comp rx=0 tx=0).stop 2023-07-04T18:52:16.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 shutdown_connections 2023-07-04T18:52:16.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f399006ace0 0x7f399006d190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f39b00c8880 0x7f39b0156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39b00cebf0 0x7f39b01e3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 --2- 172.21.15.77:0/348896554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39b015ea40 0x7f39b00c9800 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.601+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 >> 172.21.15.77:0/348896554 conn(0x7f39b006d7c0 msgr2=0x7f39b0156180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:16.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.602+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 shutdown_connections 2023-07-04T18:52:16.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.602+0000 7f39b4ba6700 1 -- 172.21.15.77:0/348896554 wait complete. 2023-07-04T18:52:16.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-04T18:51:09.218926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-04T18:51:15.974768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-04T18:52:16.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-04T18:52:16.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.739+0000 7fbb4bbc8700 1 Processor -- start 2023-07-04T18:52:16.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.739+0000 7fbb4bbc8700 1 -- start start 2023-07-04T18:52:16.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4bbc8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb440ce8c0 0x7fbb4415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4bbc8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4415f180 con 0x7fbb440d01d0 2023-07-04T18:52:16.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4bbc8700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb440cee30 con 0x7fbb441440b0 2023-07-04T18:52:16.742 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4bbc8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440cf110 con 0x7fbb440ce8c0 2023-07-04T18:52:16.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb440ce8c0 0x7fbb4415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbb440ce8c0 0x7fbb4415ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45804/0 (socket says 172.21.15.77:45804) 2023-07-04T18:52:16.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb49163700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbb440d01d0 0x7fbb440cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52940/0 (socket says 172.21.15.77:52940) 2023-07-04T18:52:16.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 -- 172.21.15.77:0/3691148790 learned_addr learned my addr 172.21.15.77:0/3691148790 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:16.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 -- 172.21.15.77:0/3691148790 >> v1:172.21.15.82:6789/0 conn(0x7fbb441440b0 legacy=0x7fbb440cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:16.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 -- 172.21.15.77:0/3691148790 >> v1:172.21.15.77:6789/0 conn(0x7fbb440d01d0 legacy=0x7fbb440cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb48962700 1 -- 172.21.15.77:0/3691148790 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717678481 0 0) 0x7fbb34001110 con 0x7fbb440d01d0 2023-07-04T18:52:16.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 -- 172.21.15.77:0/3691148790 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb440cf3f0 con 0x7fbb440ce8c0 2023-07-04T18:52:16.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb4a165700 1 --2- 172.21.15.77:0/3691148790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb440ce8c0 0x7fbb4415ec40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c009e50 tx=0x7fbb3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15312bc8c99eda7e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb48962700 1 -- 172.21.15.77:0/3691148790 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00e070 con 0x7fbb440ce8c0 2023-07-04T18:52:16.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb48962700 1 -- 172.21.15.77:0/3691148790 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c003a20 con 0x7fbb440ce8c0 2023-07-04T18:52:16.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.740+0000 7fbb48962700 1 -- 172.21.15.77:0/3691148790 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c003dd0 con 0x7fbb440ce8c0 2023-07-04T18:52:16.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/3691148790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb440ce8c0 msgr2=0x7fbb4415ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/3691148790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb440ce8c0 0x7fbb4415ec40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c009e50 tx=0x7fbb3c00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:16.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/3691148790 shutdown_connections 2023-07-04T18:52:16.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/3691148790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb440ce8c0 0x7fbb4415ec40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/3691148790 >> 172.21.15.77:0/3691148790 conn(0x7fbb4406d900 msgr2=0x7fbb4406dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/3691148790 shutdown_connections 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.741+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/3691148790 wait complete. 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.742+0000 7fbb4bbc8700 1 Processor -- start 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.742+0000 7fbb4bbc8700 1 -- start start 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbb440ce8c0 0x7fbb440c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb440d01d0 0x7fbb44157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440c9350 con 0x7fbb441440b0 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441f5d90 con 0x7fbb440d01d0 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4bbc8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb44158110 con 0x7fbb440ce8c0 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45810/0 (socket says 172.21.15.77:45810) 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 -- 172.21.15.77:0/438355332 learned_addr learned my addr 172.21.15.77:0/438355332 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:16.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb49163700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb440d01d0 0x7fbb44157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 -- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbb440ce8c0 msgr2=0x7fbb440c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbb440ce8c0 0x7fbb440c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 -- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb440d01d0 msgr2=0x7fbb44157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb440d01d0 0x7fbb44157bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 -- 172.21.15.77:0/438355332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c009b30 con 0x7fbb441440b0 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.743+0000 7fbb4a165700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbb440d1df0 tx=0x7fbb3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.744+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c004240 con 0x7fbb441440b0 2023-07-04T18:52:16.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.744+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c0043a0 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.744+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb440c9560 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.744+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c004750 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.744+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb440c9b20 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.745+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4406e330 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.745+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbb3c019070 con 0x7fbb441440b0 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.745+0000 7fbb3a7fc700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 0x7fbb2806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:16.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.746+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbb3c08a460 con 0x7fbb441440b0 2023-07-04T18:52:16.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.746+0000 7fbb49964700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 0x7fbb2806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:16.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.746+0000 7fbb49964700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 0x7fbb2806d190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbb400096f0 tx=0x7fbb40006c50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:16.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.750+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbb3c055870 con 0x7fbb441440b0 2023-07-04T18:52:16.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.934+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb4404e210 con 0x7fbb441440b0 2023-07-04T18:52:16.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.935+0000 7fbb3a7fc700 1 -- 172.21.15.77:0/438355332 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb3c058e90 con 0x7fbb441440b0 2023-07-04T18:52:16.937 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2023-07-04T18:52:16.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 msgr2=0x7fbb2806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 0x7fbb2806d190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbb400096f0 tx=0x7fbb40006c50 comp rx=0 tx=0).stop 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 msgr2=0x7fbb440a90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbb440d1df0 tx=0x7fbb3c004ad0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 shutdown_connections 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fbb2806ace0 0x7fbb2806d190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.939 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb441440b0 0x7fbb440a90c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fbb440ce8c0 0x7fbb440c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 --2- 172.21.15.77:0/438355332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb440d01d0 0x7fbb44157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 >> 172.21.15.77:0/438355332 conn(0x7fbb4406d900 msgr2=0x7fbb440c6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 shutdown_connections 2023-07-04T18:52:16.940 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:16.937+0000 7fbb4bbc8700 1 -- 172.21.15.77:0/438355332 wait complete. 2023-07-04T18:52:16.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-07-04T18:52:17.072 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.071+0000 7fba241a3700 1 Processor -- start 2023-07-04T18:52:17.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.071+0000 7fba241a3700 1 -- start start 2023-07-04T18:52:17.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba241a3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba1c0d01d0 0x7fba1c15ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba241a3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c0ce8c0 0x7fba1c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba241a3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c15f180 con 0x7fba1c1440b0 2023-07-04T18:52:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba241a3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0cee30 con 0x7fba1c0d01d0 2023-07-04T18:52:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba241a3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0cf110 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba1c0d01d0 0x7fba1c15ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba21f3f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fba1c1440b0 0x7fba1c0cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52952/0 (socket says 172.21.15.77:52952) 2023-07-04T18:52:17.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba21f3f700 1 -- 172.21.15.77:0/4253574561 learned_addr learned my addr 172.21.15.77:0/4253574561 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:17.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba22740700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c0ce8c0 0x7fba1c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 -- 172.21.15.77:0/4253574561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c0ce8c0 msgr2=0x7fba1c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c0ce8c0 0x7fba1c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 -- 172.21.15.77:0/4253574561 >> v1:172.21.15.77:6789/0 conn(0x7fba1c1440b0 legacy=0x7fba1c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 -- 172.21.15.77:0/4253574561 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0cf3f0 con 0x7fba1c0d01d0 2023-07-04T18:52:17.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba2173e700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0d01d0 0x7fba1c15ec40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fba18009840 tx=0x7fba18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753afc58122fb9ee server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba20f3d700 1 -- 172.21.15.77:0/4253574561 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18011830 con 0x7fba1c0d01d0 2023-07-04T18:52:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba20f3d700 1 -- 172.21.15.77:0/4253574561 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18011990 con 0x7fba1c0d01d0 2023-07-04T18:52:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.072+0000 7fba20f3d700 1 -- 172.21.15.77:0/4253574561 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18011d20 con 0x7fba1c0d01d0 2023-07-04T18:52:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 -- 172.21.15.77:0/4253574561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0d01d0 msgr2=0x7fba1c15ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0d01d0 0x7fba1c15ec40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fba18009840 tx=0x7fba18009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 -- 172.21.15.77:0/4253574561 shutdown_connections 2023-07-04T18:52:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c0ce8c0 0x7fba1c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 --2- 172.21.15.77:0/4253574561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0d01d0 0x7fba1c15ec40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 -- 172.21.15.77:0/4253574561 >> 172.21.15.77:0/4253574561 conn(0x7fba1c06d900 msgr2=0x7fba1c06dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 -- 172.21.15.77:0/4253574561 shutdown_connections 2023-07-04T18:52:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.073+0000 7fba241a3700 1 -- 172.21.15.77:0/4253574561 wait complete. 2023-07-04T18:52:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.074+0000 7fba241a3700 1 Processor -- start 2023-07-04T18:52:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.074+0000 7fba241a3700 1 -- start start 2023-07-04T18:52:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba1c0d01d0 0x7fba1c1ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c1440b0 0x7fba1c15ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1f07e0 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1f44f0 con 0x7fba1c0d01d0 2023-07-04T18:52:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba241a3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1f3f30 con 0x7fba1c1440b0 2023-07-04T18:52:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45820/0 (socket says 172.21.15.77:45820) 2023-07-04T18:52:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 -- 172.21.15.77:0/2959632006 learned_addr learned my addr 172.21.15.77:0/2959632006 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 -- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c1440b0 msgr2=0x7fba1c15ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba22740700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c1440b0 0x7fba1c15ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba2173e700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba1c0d01d0 0x7fba1c1ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c1440b0 0x7fba1c15ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 -- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba1c0d01d0 msgr2=0x7fba1c1ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba1c0d01d0 0x7fba1c1ef6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.075+0000 7fba21f3f700 1 -- 172.21.15.77:0/2959632006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180094a0 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba21f3f700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c480 tx=0x7fba0c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c005d70 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c0070d0 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c1f4210 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c007440 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.076+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c1f0400 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.077+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c04e210 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.077+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba0c0075a0 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.077+0000 7fba127fc700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 0x7fba0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.077+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fba0c088390 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.078+0000 7fba2173e700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 0x7fba0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.078+0000 7fba2173e700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 0x7fba0006d190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0d05c0 tx=0x7fba18008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.096 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.082+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba0c053840 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.269+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fba1c15f180 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.269+0000 7fba127fc700 1 -- 172.21.15.77:0/2959632006 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v33) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba1c15f180 con 0x7fba1c0ce8c0 2023-07-04T18:52:17.271 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2023-07-04T18:52:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.271+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 msgr2=0x7fba0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.271+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 0x7fba0006d190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0d05c0 tx=0x7fba18008030 comp rx=0 tx=0).stop 2023-07-04T18:52:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.271+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 msgr2=0x7fba1c0c71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.271+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c480 tx=0x7fba0c00c840 comp rx=0 tx=0).stop 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 shutdown_connections 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fba0006ace0 0x7fba0006d190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba1c0ce8c0 0x7fba1c0c71a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fba1c1440b0 0x7fba1c15ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 --2- 172.21.15.77:0/2959632006 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba1c0d01d0 0x7fba1c1ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 >> 172.21.15.77:0/2959632006 conn(0x7fba1c06d900 msgr2=0x7fba1c1594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 shutdown_connections 2023-07-04T18:52:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.272+0000 7fba241a3700 1 -- 172.21.15.77:0/2959632006 wait complete. 2023-07-04T18:52:17.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-04T18:52:17.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.405+0000 7f2c741cd700 1 Processor -- start 2023-07-04T18:52:17.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.405+0000 7f2c741cd700 1 -- start start 2023-07-04T18:52:17.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c741cd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c741cd700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 0x7f2c6c0cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c741cd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c15a7e0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c741cd700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0cb570 con 0x7f2c6c15e640 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c741cd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0cb290 con 0x7f2c6c15a410 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c71768700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2c6c0ca7d0 0x7f2c6c15df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:52966/0 (socket says 172.21.15.77:52966) 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c71768700 1 -- 172.21.15.77:0/1965239241 learned_addr learned my addr 172.21.15.77:0/1965239241 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c71f69700 1 --2- 172.21.15.77:0/1965239241 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c7276a700 1 --2- 172.21.15.77:0/1965239241 >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 0x7f2c6c0cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916050120 0 0) 0x7f2c6c15a7e0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.406+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c580035b0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c71f69700 1 -- 172.21.15.77:0/1965239241 >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 msgr2=0x7f2c6c0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:17.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c71f69700 1 --2- 172.21.15.77:0/1965239241 >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 0x7f2c6c0cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c71f69700 1 -- 172.21.15.77:0/1965239241 >> v1:172.21.15.77:6789/0 conn(0x7f2c6c0ca7d0 legacy=0x7f2c6c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542473279 0 0) 0x7f2c580035b0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c71f69700 1 -- 172.21.15.77:0/1965239241 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c069ee0 con 0x7f2c6c15a410 2023-07-04T18:52:17.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c7276a700 1 --2- 172.21.15.77:0/1965239241 >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 0x7f2c6c0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:17.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c71f69700 1 --2- 172.21.15.77:0/1965239241 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c06c310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009840 tx=0x7f2c5c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c0950cadf0cef9 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68001670 con 0x7f2c6c15a410 2023-07-04T18:52:17.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c004030 con 0x7f2c6c15a410 2023-07-04T18:52:17.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.407+0000 7f2c70f67700 1 -- 172.21.15.77:0/1965239241 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0043c0 con 0x7f2c6c15a410 2023-07-04T18:52:17.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 -- 172.21.15.77:0/1965239241 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 msgr2=0x7f2c6c06c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 --2- 172.21.15.77:0/1965239241 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c06c310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009840 tx=0x7f2c5c009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:17.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 -- 172.21.15.77:0/1965239241 shutdown_connections 2023-07-04T18:52:17.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 --2- 172.21.15.77:0/1965239241 >> v2:172.21.15.82:3300/0 conn(0x7f2c6c15e640 0x7f2c6c0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 --2- 172.21.15.77:0/1965239241 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c06c310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 -- 172.21.15.77:0/1965239241 >> 172.21.15.77:0/1965239241 conn(0x7f2c6c0cf220 msgr2=0x7f2c6c0cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:17.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 -- 172.21.15.77:0/1965239241 shutdown_connections 2023-07-04T18:52:17.418 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.408+0000 7f2c741cd700 1 -- 172.21.15.77:0/1965239241 wait complete. 2023-07-04T18:52:17.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.409+0000 7f2c741cd700 1 Processor -- start 2023-07-04T18:52:17.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.409+0000 7f2c741cd700 1 -- start start 2023-07-04T18:52:17.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c09e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.422 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 0x7f2c6c09bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c09f160 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c09f440 con 0x7f2c6c15e640 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c09f720 con 0x7f2c6c15a410 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45832/0 (socket says 172.21.15.77:45832) 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 -- 172.21.15.77:0/3477445166 learned_addr learned my addr 172.21.15.77:0/3477445166 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:17.423 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c7276a700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 0x7f2c6c09bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71768700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c09e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 -- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 msgr2=0x7f2c6c09e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c09e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 -- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 msgr2=0x7f2c6c09bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 0x7f2c6c09bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 -- 172.21.15.77:0/3477445166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009a70 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c7276a700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 0x7f2c6c09bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c71f69700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00a010 tx=0x7f2c5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c003e10 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c01f070 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c0094a0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c011870 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.410+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c09fce0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.411+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c04e210 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.412+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2c5c0119d0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.412+0000 7f2c627fc700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 0x7f2c5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.412+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2c5c021070 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.412+0000 7f2c71768700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 0x7f2c5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.413+0000 7f2c71768700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 0x7f2c5006d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c680011e0 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.417+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2c5c055bb0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.611+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2c6c09fed0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.611+0000 7f2c627fc700 1 -- 172.21.15.77:0/3477445166 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v33) v1 ==== 84+0+126 (secure 0 0 0) 0x7f2c5c0591d0 con 0x7f2c6c0ca7d0 2023-07-04T18:52:17.614 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:17.614 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2023-07-04T18:52:17.614 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:17.615 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2023-07-04T18:52:17.616 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2023-07-04T18:52:17.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2023-07-04T18:52:17.616 INFO:teuthology.orchestra.run.smithi077.stdout: { 2023-07-04T18:52:17.616 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 2, 2023-07-04T18:52:17.617 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": "rbd" 2023-07-04T18:52:17.617 INFO:teuthology.orchestra.run.smithi077.stdout: } 2023-07-04T18:52:17.618 INFO:teuthology.orchestra.run.smithi077.stdout:] 2023-07-04T18:52:17.619 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 msgr2=0x7f2c5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 0x7f2c5006d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c680011e0 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2023-07-04T18:52:17.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 msgr2=0x7f2c6c1532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c00a010 tx=0x7f2c5c004750 comp rx=0 tx=0).stop 2023-07-04T18:52:17.620 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 shutdown_connections 2023-07-04T18:52:17.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f2c5006ace0 0x7f2c5006d190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c6c0ca7d0 0x7f2c6c1532a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c6c15e640 0x7f2c6c09bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.621 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 --2- 172.21.15.77:0/3477445166 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f2c6c15a410 0x7f2c6c09e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.622 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 >> 172.21.15.77:0/3477445166 conn(0x7f2c6c0cf220 msgr2=0x7f2c6c15cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:17.622 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 shutdown_connections 2023-07-04T18:52:17.622 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.613+0000 7f2c741cd700 1 -- 172.21.15.77:0/3477445166 wait complete. 2023-07-04T18:52:17.627 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-04T18:52:17.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-04T18:52:17.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.749+0000 7f133d5fd700 1 Processor -- start 2023-07-04T18:52:17.751 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.749+0000 7f133d5fd700 1 -- start start 2023-07-04T18:52:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f133d5fd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f133806a550 0x7f13380b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f133d5fd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 0x7f1338160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f133d5fd700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133815ca10 con 0x7f133815c640 2023-07-04T18:52:17.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f133d5fd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13380cdc90 con 0x7f133806a550 2023-07-04T18:52:17.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f133d5fd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1338160210 con 0x7f13380aec80 2023-07-04T18:52:17.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f133806a550 0x7f13380b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f133806a550 0x7f13380b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45842/0 (socket says 172.21.15.77:45842) 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 -- 172.21.15.77:0/1325804868 learned_addr learned my addr 172.21.15.77:0/1325804868 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13377fe700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 0x7f1338160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 -- 172.21.15.77:0/1325804868 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 msgr2=0x7f1338160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 0x7f1338160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 -- 172.21.15.77:0/1325804868 >> v1:172.21.15.82:6789/0 conn(0x7f133815c640 legacy=0x7f1338069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 -- 172.21.15.77:0/1325804868 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13381604f0 con 0x7f133806a550 2023-07-04T18:52:17.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13377fe700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 0x7f1338160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f13367fc700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133806a550 0x7f13380b2260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f13280098f0 tx=0x7f1328009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dce28a47c76305 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.750+0000 7f1335ffb700 1 -- 172.21.15.77:0/1325804868 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800b8a0 con 0x7f133806a550 2023-07-04T18:52:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f1335ffb700 1 -- 172.21.15.77:0/1325804868 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800ba00 con 0x7f133806a550 2023-07-04T18:52:17.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f1335ffb700 1 -- 172.21.15.77:0/1325804868 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800bd90 con 0x7f133806a550 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 -- 172.21.15.77:0/1325804868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133806a550 msgr2=0x7f13380b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133806a550 0x7f13380b2260 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f13280098f0 tx=0x7f1328009c00 comp rx=0 tx=0).stop 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 -- 172.21.15.77:0/1325804868 shutdown_connections 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f13380aec80 0x7f1338160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 --2- 172.21.15.77:0/1325804868 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f133806a550 0x7f13380b2260 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 -- 172.21.15.77:0/1325804868 >> 172.21.15.77:0/1325804868 conn(0x7f133806bbe0 msgr2=0x7f13380b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:17.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 -- 172.21.15.77:0/1325804868 shutdown_connections 2023-07-04T18:52:17.758 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.751+0000 7f133d5fd700 1 -- 172.21.15.77:0/1325804868 wait complete. 2023-07-04T18:52:17.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.753+0000 7f133d5fd700 1 Processor -- start 2023-07-04T18:52:17.760 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.753+0000 7f133d5fd700 1 -- start start 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f133806a550 0x7f13380b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 0x7f13381466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13381462b0 con 0x7f13380aec80 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13381e7ca0 con 0x7f133806a550 2023-07-04T18:52:17.761 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1338158a50 con 0x7f133815c640 2023-07-04T18:52:17.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.762 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45848/0 (socket says 172.21.15.77:45848) 2023-07-04T18:52:17.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 -- 172.21.15.77:0/2031501477 learned_addr learned my addr 172.21.15.77:0/2031501477 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:17.763 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f1336ffd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f133806a550 0x7f13380b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13377fe700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 0x7f13381466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 -- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 msgr2=0x7f13381466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 0x7f13381466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 -- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f133806a550 msgr2=0x7f13380b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f133806a550 0x7f13380b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 -- 172.21.15.77:0/2031501477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328009550 con 0x7f13380aec80 2023-07-04T18:52:17.764 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13377fe700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 0x7f13381466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f13367fc700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1328009130 tx=0x7f1328004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280036a0 con 0x7f13380aec80 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132801e070 con 0x7f13380aec80 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328003a40 con 0x7f13380aec80 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.754+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338145630 con 0x7f13380aec80 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.755+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1338145bf0 con 0x7f13380aec80 2023-07-04T18:52:17.765 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.755+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13381463f0 con 0x7f13380aec80 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.756+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1328003ba0 con 0x7f13380aec80 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.756+0000 7f131f7fe700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 0x7f132406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.756+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1328020070 con 0x7f13380aec80 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.756+0000 7f1336ffd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 0x7f132406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.757+0000 7f1336ffd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 0x7f132406d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f132c0016c0 tx=0x7f132c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:17.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.761+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f13381463f0 con 0x7f13380aec80 2023-07-04T18:52:17.947 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:17.946+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f13381463f0 con 0x7f13380aec80 2023-07-04T18:52:18.440 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.438+0000 7f131f7fe700 1 -- 172.21.15.77:0/2031501477 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v34) v1 ==== 138+0+0 (secure 0 0 0) 0x7f13381463f0 con 0x7f13380aec80 2023-07-04T18:52:18.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.441+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 msgr2=0x7f132406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:18.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.441+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 0x7f132406d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f132c0016c0 tx=0x7f132c009040 comp rx=0 tx=0).stop 2023-07-04T18:52:18.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.441+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 msgr2=0x7f13381447e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:18.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.441+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1328009130 tx=0x7f1328004750 comp rx=0 tx=0).stop 2023-07-04T18:52:18.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 shutdown_connections 2023-07-04T18:52:18.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f132406ace0 0x7f132406d190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f133815c640 0x7f13381466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.444 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f133806a550 0x7f13380b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 --2- 172.21.15.77:0/2031501477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13380aec80 0x7f13381447e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.442+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 >> 172.21.15.77:0/2031501477 conn(0x7f133806bbe0 msgr2=0x7f13380705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:18.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.443+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 shutdown_connections 2023-07-04T18:52:18.445 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.443+0000 7f133d5fd700 1 -- 172.21.15.77:0/2031501477 wait complete. 2023-07-04T18:52:18.448 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2023-07-04T18:52:18.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-04T18:52:18.586 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.585+0000 7fcfb9a65700 1 Processor -- start 2023-07-04T18:52:18.587 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.585+0000 7fcfb9a65700 1 -- start start 2023-07-04T18:52:18.588 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfb415a430 0x7fcfb40ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fcfb40cebf0 0x7fcfb40cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb415ea40 0x7fcfb40cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb40cfe60 con 0x7fcfb415a430 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb40cf100 con 0x7fcfb40cebf0 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb9a65700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb415df60 con 0x7fcfb415ea40 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfb415a430 0x7fcfb40ce6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.589 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcfb415a430 0x7fcfb40ce6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45854/0 (socket says 172.21.15.77:45854) 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 -- 172.21.15.77:0/3067646139 learned_addr learned my addr 172.21.15.77:0/3067646139 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb2ffd700 1 --2- 172.21.15.77:0/3067646139 >> v2:172.21.15.82:3300/0 conn(0x7fcfb40cebf0 0x7fcfb40cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb3fff700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb415ea40 0x7fcfb40cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 -- 172.21.15.77:0/3067646139 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb415ea40 msgr2=0x7fcfb40cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb415ea40 0x7fcfb40cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 -- 172.21.15.77:0/3067646139 >> v2:172.21.15.82:3300/0 conn(0x7fcfb40cebf0 msgr2=0x7fcfb40cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:18.590 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/3067646139 >> v2:172.21.15.82:3300/0 conn(0x7fcfb40cebf0 0x7fcfb40cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 -- 172.21.15.77:0/3067646139 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb415e240 con 0x7fcfb415a430 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.586+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415a430 0x7fcfb40ce6b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40713a0 tx=0x7fcfa8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d50c99150801848 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb27fc700 1 -- 172.21.15.77:0/3067646139 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8011830 con 0x7fcfb415a430 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb27fc700 1 -- 172.21.15.77:0/3067646139 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa8011990 con 0x7fcfb415a430 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb27fc700 1 -- 172.21.15.77:0/3067646139 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8011d20 con 0x7fcfb415a430 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 -- 172.21.15.77:0/3067646139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415a430 msgr2=0x7fcfb40ce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415a430 0x7fcfb40ce6b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40713a0 tx=0x7fcfa8009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.591 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 -- 172.21.15.77:0/3067646139 shutdown_connections 2023-07-04T18:52:18.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb415ea40 0x7fcfb40cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/3067646139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415a430 0x7fcfb40ce6b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/3067646139 >> v2:172.21.15.82:3300/0 conn(0x7fcfb40cebf0 0x7fcfb40cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.592 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 -- 172.21.15.77:0/3067646139 >> 172.21.15.77:0/3067646139 conn(0x7fcfb406d7c0 msgr2=0x7fcfb406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 -- 172.21.15.77:0/3067646139 shutdown_connections 2023-07-04T18:52:18.593 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.587+0000 7fcfb9a65700 1 -- 172.21.15.77:0/3067646139 wait complete. 2023-07-04T18:52:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.588+0000 7fcfb9a65700 1 Processor -- start 2023-07-04T18:52:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 -- start start 2023-07-04T18:52:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfb40cebf0 0x7fcfb4154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb40afea0 0x7fcfb40b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4154680 con 0x7fcfb415ea40 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40b0fc0 con 0x7fcfb40cebf0 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb9a65700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40b03b0 con 0x7fcfb40afea0 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45860/0 (socket says 172.21.15.77:45860) 2023-07-04T18:52:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.589+0000 7fcfb2ffd700 1 -- 172.21.15.77:0/504305515 learned_addr learned my addr 172.21.15.77:0/504305515 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb3fff700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb40afea0 0x7fcfb40b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfb40cebf0 0x7fcfb4154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 -- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb40afea0 msgr2=0x7fcfb40b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb40afea0 0x7fcfb40b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 -- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfb40cebf0 msgr2=0x7fcfb4154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfb40cebf0 0x7fcfb4154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 -- 172.21.15.77:0/504305515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8009550 con 0x7fcfb415ea40 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb2ffd700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9b0 tx=0x7fcfa400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4009710 con 0x7fcfb415ea40 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa400c850 con 0x7fcfb415ea40 2023-07-04T18:52:18.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb40b0690 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa400cbc0 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.590+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb40b0c50 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.591+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb40c8c90 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.591+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcfa400cd20 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.592+0000 7fcf9bfff700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.592+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fcfa40893a0 con 0x7fcfb415ea40 2023-07-04T18:52:18.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.592+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:18.602 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.592+0000 7fcfb37fe700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 0x7fcfa006d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009c10 tx=0x7fcfa8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:18.602 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.596+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcfb40c8c90 con 0x7fcfb415ea40 2023-07-04T18:52:18.785 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:18.784+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcfb40c8c90 con 0x7fcfb415ea40 2023-07-04T18:52:19.443 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.441+0000 7fcf9bfff700 1 -- 172.21.15.77:0/504305515 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v35) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcfb40c8c90 con 0x7fcfb415ea40 2023-07-04T18:52:19.447 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 msgr2=0x7fcfa006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 0x7fcfa006d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009c10 tx=0x7fcfa8008030 comp rx=0 tx=0).stop 2023-07-04T18:52:19.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 msgr2=0x7fcfb40af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.448 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400e9b0 tx=0x7fcfa400ed70 comp rx=0 tx=0).stop 2023-07-04T18:52:19.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 shutdown_connections 2023-07-04T18:52:19.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fcfa006ace0 0x7fcfa006d190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.449 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fcfb40afea0 0x7fcfb40b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfb40cebf0 0x7fcfb4154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 --2- 172.21.15.77:0/504305515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfb415ea40 0x7fcfb40af960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.450 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 >> 172.21.15.77:0/504305515 conn(0x7fcfb406d7c0 msgr2=0x7fcfb40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:19.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 shutdown_connections 2023-07-04T18:52:19.452 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.445+0000 7fcfb9a65700 1 -- 172.21.15.77:0/504305515 wait complete. 2023-07-04T18:52:19.454 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2023-07-04T18:52:19.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-04T18:52:19.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.592+0000 7fa874134700 1 Processor -- start 2023-07-04T18:52:19.594 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.592+0000 7fa874134700 1 -- start start 2023-07-04T18:52:19.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa874134700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa86c069e40 0x7fa86c06a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.596 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa874134700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 0x7fa86c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa874134700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86c0cf750 con 0x7fa86c0d01c0 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa874134700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c0af060 con 0x7fa86c069e40 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa874134700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86c0cf9c0 con 0x7fa86c0aeb40 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa871ed0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa86c069e40 0x7fa86c06a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa8716cf700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa86c0d01c0 0x7fa86c0ae430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53004/0 (socket says 172.21.15.77:53004) 2023-07-04T18:52:19.597 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa871ed0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa86c069e40 0x7fa86c06a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45866/0 (socket says 172.21.15.77:45866) 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.593+0000 7fa8716cf700 1 -- 172.21.15.77:0/2959730274 learned_addr learned my addr 172.21.15.77:0/2959730274 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa8726d1700 1 --2- 172.21.15.77:0/2959730274 >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 0x7fa86c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa871ed0700 1 -- 172.21.15.77:0/2959730274 >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 msgr2=0x7fa86c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa871ed0700 1 --2- 172.21.15.77:0/2959730274 >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 0x7fa86c15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa871ed0700 1 -- 172.21.15.77:0/2959730274 >> v1:172.21.15.77:6789/0 conn(0x7fa86c0d01c0 legacy=0x7fa86c0ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa871ed0700 1 -- 172.21.15.77:0/2959730274 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0cfca0 con 0x7fa86c069e40 2023-07-04T18:52:19.598 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa8726d1700 1 --2- 172.21.15.77:0/2959730274 >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 0x7fa86c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:19.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa871ed0700 1 --2- 172.21.15.77:0/2959730274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c069e40 0x7fa86c06a210 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa86c071940 tx=0x7fa85c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0b7bf367c2df09 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:19.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa870ece700 1 -- 172.21.15.77:0/2959730274 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c011830 con 0x7fa86c069e40 2023-07-04T18:52:19.599 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa870ece700 1 -- 172.21.15.77:0/2959730274 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c011990 con 0x7fa86c069e40 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa870ece700 1 -- 172.21.15.77:0/2959730274 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c011d20 con 0x7fa86c069e40 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa874134700 1 -- 172.21.15.77:0/2959730274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c069e40 msgr2=0x7fa86c06a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.594+0000 7fa874134700 1 --2- 172.21.15.77:0/2959730274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c069e40 0x7fa86c06a210 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa86c071940 tx=0x7fa85c009b20 comp rx=0 tx=0).stop 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 -- 172.21.15.77:0/2959730274 shutdown_connections 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 --2- 172.21.15.77:0/2959730274 >> v2:172.21.15.82:3300/0 conn(0x7fa86c0aeb40 0x7fa86c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 --2- 172.21.15.77:0/2959730274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c069e40 0x7fa86c06a210 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.600 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 -- 172.21.15.77:0/2959730274 >> 172.21.15.77:0/2959730274 conn(0x7fa86c06af50 msgr2=0x7fa86c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:19.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 -- 172.21.15.77:0/2959730274 shutdown_connections 2023-07-04T18:52:19.601 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.595+0000 7fa874134700 1 -- 172.21.15.77:0/2959730274 wait complete. 2023-07-04T18:52:19.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.596+0000 7fa874134700 1 Processor -- start 2023-07-04T18:52:19.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.596+0000 7fa874134700 1 -- start start 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa86c069e40 0x7fa86c1414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa86c0d01c0 0x7fa86c0b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1419e0 con 0x7fa86c0aeb40 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f2790 con 0x7fa86c069e40 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1f2a70 con 0x7fa86c0d01c0 2023-07-04T18:52:19.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45874/0 (socket says 172.21.15.77:45874) 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 -- 172.21.15.77:0/2373831291 learned_addr learned my addr 172.21.15.77:0/2373831291 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa871ed0700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa86c069e40 0x7fa86c1414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8726d1700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa86c0d01c0 0x7fa86c0b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 -- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa86c0d01c0 msgr2=0x7fa86c0b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa86c0d01c0 0x7fa86c0b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 -- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa86c069e40 msgr2=0x7fa86c1414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa86c069e40 0x7fa86c1414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 -- 172.21.15.77:0/2373831291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0094a0 con 0x7fa86c0aeb40 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8716cf700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c520 tx=0x7fa86800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868005d70 con 0x7fa86c0aeb40 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8680070d0 con 0x7fa86c0aeb40 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c1f2d50 con 0x7fa86c0aeb40 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868007440 con 0x7fa86c0aeb40 2023-07-04T18:52:19.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.597+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1f3310 con 0x7fa86c0aeb40 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.598+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c153b10 con 0x7fa86c0aeb40 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.599+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa8680075a0 con 0x7fa86c0aeb40 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.599+0000 7fa8627fc700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 0x7fa85006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.599+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fa868088700 con 0x7fa86c0aeb40 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.599+0000 7fa871ed0700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 0x7fa85006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:19.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.600+0000 7fa871ed0700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 0x7fa85006d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0090b0 tx=0x7fa85c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:19.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.604+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa86c153b10 con 0x7fa86c0aeb40 2023-07-04T18:52:19.841 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:19.839+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa86c153b10 con 0x7fa86c0aeb40 2023-07-04T18:52:20.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.475+0000 7fa8627fc700 1 -- 172.21.15.77:0/2373831291 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa86c153b10 con 0x7fa86c0aeb40 2023-07-04T18:52:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 msgr2=0x7fa85006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 0x7fa85006d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0090b0 tx=0x7fa85c01c040 comp rx=0 tx=0).stop 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 msgr2=0x7fa86c14be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c520 tx=0x7fa86800c830 comp rx=0 tx=0).stop 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 shutdown_connections 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fa85006ace0 0x7fa85006d190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fa86c0d01c0 0x7fa86c0b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa86c069e40 0x7fa86c1414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 --2- 172.21.15.77:0/2373831291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa86c0aeb40 0x7fa86c14be10 secure :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c520 tx=0x7fa86800c830 comp rx=0 tx=0).stop 2023-07-04T18:52:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.483+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 >> 172.21.15.77:0/2373831291 conn(0x7fa86c06af50 msgr2=0x7fa86c06dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.484+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 shutdown_connections 2023-07-04T18:52:20.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.486+0000 7fa874134700 1 -- 172.21.15.77:0/2373831291 wait complete. 2023-07-04T18:52:20.489 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 5 and data pool 6 2023-07-04T18:52:20.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:20.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.626+0000 7f7070bb8700 1 Processor -- start 2023-07-04T18:52:20.628 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.626+0000 7f7070bb8700 1 -- start start 2023-07-04T18:52:20.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7070bb8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f706c15ca10 0x7f706c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7070bb8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c0cf830 con 0x7f706c070db0 2023-07-04T18:52:20.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7070bb8700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c0711d0 con 0x7f706c15c640 2023-07-04T18:52:20.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7070bb8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0717f0 con 0x7f706c15ca10 2023-07-04T18:52:20.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7069d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f706c15ca10 0x7f706c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f706ad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f706c070db0 0x7f706c0cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53018/0 (socket says 172.21.15.77:53018) 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f706ad9d700 1 -- 172.21.15.77:0/2038085293 learned_addr learned my addr 172.21.15.77:0/2038085293 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7069d9b700 1 -- 172.21.15.77:0/2038085293 >> v1:172.21.15.82:6789/0 conn(0x7f706c15c640 legacy=0x7f706c0cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7069d9b700 1 -- 172.21.15.77:0/2038085293 >> v1:172.21.15.77:6789/0 conn(0x7f706c070db0 legacy=0x7f706c0cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7069d9b700 1 -- 172.21.15.77:0/2038085293 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0cd290 con 0x7f706c15ca10 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f7069d9b700 1 --2- 172.21.15.77:0/2038085293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c15ca10 0x7f706c070870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7060009840 tx=0x7f7060009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafb704a97285834 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.627+0000 7f706959a700 1 -- 172.21.15.77:0/2038085293 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060004230 con 0x7f706c15ca10 2023-07-04T18:52:20.638 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f706959a700 1 -- 172.21.15.77:0/2038085293 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060004390 con 0x7f706c15ca10 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f706959a700 1 -- 172.21.15.77:0/2038085293 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70600036a0 con 0x7f706c15ca10 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 -- 172.21.15.77:0/2038085293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c15ca10 msgr2=0x7f706c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2038085293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c15ca10 0x7f706c070870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7060009840 tx=0x7f7060009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 -- 172.21.15.77:0/2038085293 shutdown_connections 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2038085293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c15ca10 0x7f706c070870 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 -- 172.21.15.77:0/2038085293 >> 172.21.15.77:0/2038085293 conn(0x7f706c06bba0 msgr2=0x7f706c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 -- 172.21.15.77:0/2038085293 shutdown_connections 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.628+0000 7f7070bb8700 1 -- 172.21.15.77:0/2038085293 wait complete. 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.629+0000 7f7070bb8700 1 Processor -- start 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.629+0000 7f7070bb8700 1 -- start start 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 0x7f706c0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f706c15ca10 0x7f706c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c8290 con 0x7f706c070db0 2023-07-04T18:52:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c7e00 con 0x7f706c15c640 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f7070bb8700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0c6f50 con 0x7f706c15ca10 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f706a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f706a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45886/0 (socket says 172.21.15.77:45886) 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.630+0000 7f706a59c700 1 -- 172.21.15.77:0/2553328953 learned_addr learned my addr 172.21.15.77:0/2553328953 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f7069d9b700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 0x7f706c0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706ad9d700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f706c15ca10 0x7f706c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 -- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f706c15ca10 msgr2=0x7f706c153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f706c15ca10 0x7f706c153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 -- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 msgr2=0x7f706c0b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 0x7f706c0b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 -- 172.21.15.77:0/2553328953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70600094a0 con 0x7f706c070db0 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f7069d9b700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 0x7f706c0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f706a59c700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f705c00b4a0 tx=0x7f705c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c011830 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c011990 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c0b37a0 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c011d40 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.631+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c0c72a0 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.632+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c04e210 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.632+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f705c01a400 con 0x7f706c070db0 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.633+0000 7f705affd700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 0x7f704c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.633+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f705c089200 con 0x7f706c070db0 2023-07-04T18:52:20.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.633+0000 7f7069d9b700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 0x7f704c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.633+0000 7f7069d9b700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 0x7f704c06d190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f706c0addf0 tx=0x7f7060008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.637+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f705c0542b0 con 0x7f706c070db0 2023-07-04T18:52:20.828 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.826+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f706c0b5880 con 0x7f706c070db0 2023-07-04T18:52:20.829 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.828+0000 7f705affd700 1 -- 172.21.15.77:0/2553328953 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+18328 (secure 0 0 0) 0x7f705c0578d0 con 0x7f706c070db0 2023-07-04T18:52:20.829 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:20.830 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:19.841030+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:52:17.964674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-07-04T18:52:18.802651+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:20.831 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 msgr2=0x7f704c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 0x7f704c06d190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f706c0addf0 tx=0x7f7060008030 comp rx=0 tx=0).stop 2023-07-04T18:52:20.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 msgr2=0x7f706c0b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.832 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f705c00b4a0 tx=0x7f705c00b860 comp rx=0 tx=0).stop 2023-07-04T18:52:20.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 shutdown_connections 2023-07-04T18:52:20.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f704c06ace0 0x7f704c06d190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f706c070db0 0x7f706c0b3260 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f706c15ca10 0x7f706c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.833 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 --2- 172.21.15.77:0/2553328953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f706c15c640 0x7f706c0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.834 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 >> 172.21.15.77:0/2553328953 conn(0x7f706c06bba0 msgr2=0x7f706c0ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:20.835 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 shutdown_connections 2023-07-04T18:52:20.835 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.830+0000 7f7070bb8700 1 -- 172.21.15.77:0/2553328953 wait complete. 2023-07-04T18:52:20.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.963+0000 7f7c8ade3700 1 Processor -- start 2023-07-04T18:52:20.965 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.963+0000 7f7c8ade3700 1 -- start start 2023-07-04T18:52:20.966 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c8ade3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c840aef10 0x7f7c84069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c8ade3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c84071150 0x7f7c8406a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c8ade3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c840cb6d0 con 0x7f7c840716a0 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c8ade3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c840af2e0 con 0x7f7c84071150 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c8ade3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c84070d80 con 0x7f7c840aef10 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c89380700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7c84071150 0x7f7c8406a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c83fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7c840716a0 0x7f7c8415dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53034/0 (socket says 172.21.15.77:53034) 2023-07-04T18:52:20.967 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.964+0000 7f7c83fff700 1 -- 172.21.15.77:0/1888417528 learned_addr learned my addr 172.21.15.77:0/1888417528 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c88b7f700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c840aef10 0x7f7c84069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c89380700 1 -- 172.21.15.77:0/1888417528 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c840aef10 msgr2=0x7f7c84069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c89380700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c840aef10 0x7f7c84069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c89380700 1 -- 172.21.15.77:0/1888417528 >> v1:172.21.15.77:6789/0 conn(0x7f7c840716a0 legacy=0x7f7c8415dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c89380700 1 -- 172.21.15.77:0/1888417528 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c840cb080 con 0x7f7c84071150 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c837fe700 1 -- 172.21.15.77:0/1888417528 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296250035 0 0) 0x7f7c840cb6d0 con 0x7f7c840716a0 2023-07-04T18:52:20.968 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c89380700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c84071150 0x7f7c8406a380 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009d90 tx=0x7f7c7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69571e5d07ba122e server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c837fe700 1 -- 172.21.15.77:0/1888417528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c00e070 con 0x7f7c84071150 2023-07-04T18:52:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c837fe700 1 -- 172.21.15.77:0/1888417528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c00bbd0 con 0x7f7c84071150 2023-07-04T18:52:20.969 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c837fe700 1 -- 172.21.15.77:0/1888417528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c012430 con 0x7f7c84071150 2023-07-04T18:52:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c8ade3700 1 -- 172.21.15.77:0/1888417528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c84071150 msgr2=0x7f7c8406a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.965+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c84071150 0x7f7c8406a380 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009d90 tx=0x7f7c7c00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:20.971 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 -- 172.21.15.77:0/1888417528 shutdown_connections 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c84071150 0x7f7c8406a380 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/1888417528 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c840aef10 0x7f7c84069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 -- 172.21.15.77:0/1888417528 >> 172.21.15.77:0/1888417528 conn(0x7f7c8406bba0 msgr2=0x7f7c840b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 -- 172.21.15.77:0/1888417528 shutdown_connections 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.966+0000 7f7c8ade3700 1 -- 172.21.15.77:0/1888417528 wait complete. 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.967+0000 7f7c8ade3700 1 Processor -- start 2023-07-04T18:52:20.972 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.967+0000 7f7c8ade3700 1 -- start start 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c84071150 0x7f7c84150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c840716a0 0x7f7c840ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8415d3e0 con 0x7f7c840aef10 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c840ad360 con 0x7f7c840716a0 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c840ad640 con 0x7f7c84071150 2023-07-04T18:52:20.973 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45904/0 (socket says 172.21.15.77:45904) 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 -- 172.21.15.77:0/2372354552 learned_addr learned my addr 172.21.15.77:0/2372354552 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c83fff700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c840716a0 0x7f7c840ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c88b7f700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c84071150 0x7f7c84150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 -- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c84071150 msgr2=0x7f7c84150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c84071150 0x7f7c84150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 -- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c840716a0 msgr2=0x7f7c840ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:20.974 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c840716a0 0x7f7c840ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 -- 172.21.15.77:0/2372354552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c009a70 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c89380700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c013000 tx=0x7f7c7c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c012430 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7c020070 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c840ad920 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c004280 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.968+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c84151440 con 0x7f7c840aef10 2023-07-04T18:52:20.975 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.969+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c840adde0 con 0x7f7c840aef10 2023-07-04T18:52:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.970+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7c7c0043e0 con 0x7f7c840aef10 2023-07-04T18:52:20.976 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.970+0000 7f7c817fa700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 0x7f7c6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:20.977 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.970+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f7c7c022070 con 0x7f7c840aef10 2023-07-04T18:52:20.977 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.970+0000 7f7c88b7f700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 0x7f7c6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:20.977 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.971+0000 7f7c88b7f700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 0x7f7c6806d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7c78006b20 tx=0x7f7c78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:20.977 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:20.975+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7c7c056680 con 0x7f7c840aef10 2023-07-04T18:52:21.161 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.160+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c8404e210 con 0x7f7c840aef10 2023-07-04T18:52:21.163 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.161+0000 7f7c817fa700 1 -- 172.21.15.77:0/2372354552 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+18328 (secure 0 0 0) 0x7f7c7c059ca0 con 0x7f7c840aef10 2023-07-04T18:52:21.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:21.165 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:19.841030+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:52:17.964674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-07-04T18:52:18.802651+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:21.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 msgr2=0x7f7c6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 0x7f7c6806d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7c78006b20 tx=0x7f7c78008040 comp rx=0 tx=0).stop 2023-07-04T18:52:21.167 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 msgr2=0x7f7c840ceb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c013000 tx=0x7f7c7c004ad0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 shutdown_connections 2023-07-04T18:52:21.168 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.163+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f7c6806ace0 0x7f7c6806d190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c840aef10 0x7f7c840ceb60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f7c84071150 0x7f7c84150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 --2- 172.21.15.77:0/2372354552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c840716a0 0x7f7c840ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.169 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 >> 172.21.15.77:0/2372354552 conn(0x7f7c8406bba0 msgr2=0x7f7c840b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 shutdown_connections 2023-07-04T18:52:21.170 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.164+0000 7f7c8ade3700 1 -- 172.21.15.77:0/2372354552 wait complete. 2023-07-04T18:52:21.182 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-04T18:52:21.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.303+0000 7f98e8ebd700 1 Processor -- start 2023-07-04T18:52:21.305 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.303+0000 7f98e8ebd700 1 -- start start 2023-07-04T18:52:21.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e8ebd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e41590b0 0x7f98e415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.307 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e8ebd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98e406cfc0 0x7f98e406d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e8ebd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40b3f60 con 0x7f98e40b3330 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e8ebd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e406caa0 con 0x7f98e406cfc0 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e8ebd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40b4b00 con 0x7f98e41590b0 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e1d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f98e40b3330 0x7f98e40b43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53046/0 (socket says 172.21.15.77:53046) 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e1d9b700 1 -- 172.21.15.77:0/2338838778 learned_addr learned my addr 172.21.15.77:0/2338838778 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.304+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/2338838778 >> v2:172.21.15.77:3300/0 conn(0x7f98e406cfc0 0x7f98e406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e259c700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e41590b0 0x7f98e415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e2d9d700 1 -- 172.21.15.77:0/2338838778 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e41590b0 msgr2=0x7f98e415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e41590b0 0x7f98e415d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e2d9d700 1 -- 172.21.15.77:0/2338838778 >> v1:172.21.15.77:6789/0 conn(0x7f98e40b3330 legacy=0x7f98e40b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e159a700 1 -- 172.21.15.77:0/2338838778 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119034949 0 0) 0x7f98e40b3f60 con 0x7f98e40b3330 2023-07-04T18:52:21.309 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e2d9d700 1 -- 172.21.15.77:0/2338838778 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e406d940 con 0x7f98e406cfc0 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e406cfc0 0x7f98e406d400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009d90 tx=0x7f98dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f03cb56c828aa64 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e159a700 1 -- 172.21.15.77:0/2338838778 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00e070 con 0x7f98e406cfc0 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e159a700 1 -- 172.21.15.77:0/2338838778 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc00bbd0 con 0x7f98e406cfc0 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e159a700 1 -- 172.21.15.77:0/2338838778 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc012430 con 0x7f98e406cfc0 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e8ebd700 1 -- 172.21.15.77:0/2338838778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e406cfc0 msgr2=0x7f98e406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.305+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e406cfc0 0x7f98e406d400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009d90 tx=0x7f98dc00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 -- 172.21.15.77:0/2338838778 shutdown_connections 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e406cfc0 0x7f98e406d400 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/2338838778 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e41590b0 0x7f98e415d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 -- 172.21.15.77:0/2338838778 >> 172.21.15.77:0/2338838778 conn(0x7f98e406af50 msgr2=0x7f98e406b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 -- 172.21.15.77:0/2338838778 shutdown_connections 2023-07-04T18:52:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.306+0000 7f98e8ebd700 1 -- 172.21.15.77:0/2338838778 wait complete. 2023-07-04T18:52:21.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.307+0000 7f98e8ebd700 1 Processor -- start 2023-07-04T18:52:21.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.307+0000 7f98e8ebd700 1 -- start start 2023-07-04T18:52:21.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e406cfc0 0x7f98e40ab370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e40b3330 0x7f98e40ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40acbc0 con 0x7f98e41590b0 2023-07-04T18:52:21.316 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40acea0 con 0x7f98e406cfc0 2023-07-04T18:52:21.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40ad180 con 0x7f98e40b3330 2023-07-04T18:52:21.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45918/0 (socket says 172.21.15.77:45918) 2023-07-04T18:52:21.317 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 -- 172.21.15.77:0/4035757129 learned_addr learned my addr 172.21.15.77:0/4035757129 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e259c700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e406cfc0 0x7f98e40ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e1d9b700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e40b3330 0x7f98e40ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 -- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e40b3330 msgr2=0x7f98e40ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e40b3330 0x7f98e40ac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 -- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e406cfc0 msgr2=0x7f98e40ab370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:21.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e406cfc0 0x7f98e40ab370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 -- 172.21.15.77:0/4035757129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc009a70 con 0x7f98e41590b0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e2d9d700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f98e4071970 tx=0x7f98dc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc012430 con 0x7f98e41590b0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc020070 con 0x7f98e41590b0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e40ad460 con 0x7f98e41590b0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.308+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc004280 con 0x7f98e41590b0 2023-07-04T18:52:21.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.309+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e40ac8a0 con 0x7f98e41590b0 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.309+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e40ad980 con 0x7f98e41590b0 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.310+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f98dc0043e0 con 0x7f98e41590b0 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.310+0000 7f98d2ffd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 0x7f98c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.310+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f98dc022070 con 0x7f98e41590b0 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.311+0000 7f98e259c700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 0x7f98c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:21.320 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.311+0000 7f98e259c700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 0x7f98c406d190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006b20 tx=0x7f98d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:21.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:52:21.321 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.315+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f98e40ad980 con 0x7f98e41590b0 2023-07-04T18:52:21.323 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:52:21.561 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:21.559+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f98e40ad980 con 0x7f98e41590b0 2023-07-04T18:52:22.425 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T18:52:22.423+0000 7fa6f8ddd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-04T18:52:22.482 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.481+0000 7f98d2ffd700 1 -- 172.21.15.77:0/4035757129 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f98e40ad980 con 0x7f98e41590b0 2023-07-04T18:52:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.483+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 msgr2=0x7f98c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:22.485 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.483+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 0x7f98c406d190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006b20 tx=0x7f98d8008040 comp rx=0 tx=0).stop 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 msgr2=0x7f98e41504b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f98e4071970 tx=0x7f98dc004ad0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 shutdown_connections 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f98c406ace0 0x7f98c406d190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e41590b0 0x7f98e41504b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e406cfc0 0x7f98e40ab370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.486 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 --2- 172.21.15.77:0/4035757129 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f98e40b3330 0x7f98e40ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 >> 172.21.15.77:0/4035757129 conn(0x7f98e406af50 msgr2=0x7f98e4153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 shutdown_connections 2023-07-04T18:52:22.487 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.484+0000 7f98e8ebd700 1 -- 172.21.15.77:0/4035757129 wait complete. 2023-07-04T18:52:22.500 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-07-04T18:52:22.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-07-04T18:52:22.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.627+0000 7f68924b6700 1 Processor -- start 2023-07-04T18:52:22.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.627+0000 7f68924b6700 1 -- start start 2023-07-04T18:52:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 0x7f688c06a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f688c0d01c0 0x7f688c0ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 0x7f688c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c0cf750 con 0x7f688c0d01c0 2023-07-04T18:52:22.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c0aedb0 con 0x7f688c069e40 2023-07-04T18:52:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f68924b6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0af060 con 0x7f688c0ae970 2023-07-04T18:52:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f688b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f688c0d01c0 0x7f688c0ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f688b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f688c0d01c0 0x7f688c0ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45922/0 (socket says 172.21.15.77:45922) 2023-07-04T18:52:22.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f688b7fe700 1 -- 172.21.15.77:0/924753173 learned_addr learned my addr 172.21.15.77:0/924753173 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f6890a53700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 0x7f688c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.628+0000 7f688bfff700 1 --2- 172.21.15.77:0/924753173 >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 0x7f688c06a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 -- 172.21.15.77:0/924753173 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 msgr2=0x7f688c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 0x7f688c15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.645 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 -- 172.21.15.77:0/924753173 >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 msgr2=0x7f688c06a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 --2- 172.21.15.77:0/924753173 >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 0x7f688c06a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 -- 172.21.15.77:0/924753173 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0cf9c0 con 0x7f688c0d01c0 2023-07-04T18:52:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f6890a53700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 0x7f688c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:22.646 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688bfff700 1 --2- 172.21.15.77:0/924753173 >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 0x7f688c06a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688b7fe700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0d01c0 0x7f688c0ae430 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f688c0712a0 tx=0x7f687c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc882b93edacc77b server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688affd700 1 -- 172.21.15.77:0/924753173 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c00b870 con 0x7f688c0d01c0 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688affd700 1 -- 172.21.15.77:0/924753173 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c00b9d0 con 0x7f688c0d01c0 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f688affd700 1 -- 172.21.15.77:0/924753173 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c00bd60 con 0x7f688c0d01c0 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 -- 172.21.15.77:0/924753173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0d01c0 msgr2=0x7f688c0ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0d01c0 0x7f688c0ae430 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f688c0712a0 tx=0x7f687c009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.647 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 -- 172.21.15.77:0/924753173 shutdown_connections 2023-07-04T18:52:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0ae970 0x7f688c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 --2- 172.21.15.77:0/924753173 >> v2:172.21.15.82:3300/0 conn(0x7f688c069e40 0x7f688c06a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.648 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 --2- 172.21.15.77:0/924753173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0d01c0 0x7f688c0ae430 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.629+0000 7f68924b6700 1 -- 172.21.15.77:0/924753173 >> 172.21.15.77:0/924753173 conn(0x7f688c06af50 msgr2=0x7f688c06b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:22.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.630+0000 7f68924b6700 1 -- 172.21.15.77:0/924753173 shutdown_connections 2023-07-04T18:52:22.649 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.630+0000 7f68924b6700 1 -- 172.21.15.77:0/924753173 wait complete. 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.631+0000 7f68924b6700 1 Processor -- start 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.631+0000 7f68924b6700 1 -- start start 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 0x7f688c1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 0x7f688c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1411c0 con 0x7f688c0ae970 2023-07-04T18:52:22.650 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c141430 con 0x7f688c069e40 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c141710 con 0x7f688c0d01c0 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45930/0 (socket says 172.21.15.77:45930) 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 -- 172.21.15.77:0/982511768 learned_addr learned my addr 172.21.15.77:0/982511768 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688bfff700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 0x7f688c1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.651 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f6890a53700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 0x7f688c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 -- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 msgr2=0x7f688c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 0x7f688c0b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 -- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 msgr2=0x7f688c1f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 0x7f688c1f4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 -- 172.21.15.77:0/982511768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c009550 con 0x7f688c0ae970 2023-07-04T18:52:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688bfff700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 0x7f688c1f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:22.653 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f6890a53700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 0x7f688c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f688b7fe700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f687c009c10 tx=0x7f687c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0036a0 con 0x7f688c0ae970 2023-07-04T18:52:22.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c003800 con 0x7f688c0ae970 2023-07-04T18:52:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c1419f0 con 0x7f688c0ae970 2023-07-04T18:52:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c003bb0 con 0x7f688c0ae970 2023-07-04T18:52:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.632+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c1f31b0 con 0x7f688c0ae970 2023-07-04T18:52:22.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.633+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c141c00 con 0x7f688c0ae970 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.634+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f687c003d10 con 0x7f688c0ae970 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.634+0000 7f6888ff9700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.634+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f687c08ac10 con 0x7f688c0ae970 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.634+0000 7f688bfff700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.635+0000 7f688bfff700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 0x7f687806d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f688c156430 tx=0x7f6880008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:22.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.638+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f687c055e70 con 0x7f688c0ae970 2023-07-04T18:52:22.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:22.867+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f688c04e210 con 0x7f688c0ae970 2023-07-04T18:52:23.488 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.486+0000 7f6888ff9700 1 -- 172.21.15.77:0/982511768 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f687c0150c0 con 0x7f688c0ae970 2023-07-04T18:52:23.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 msgr2=0x7f687806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 0x7f687806d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f688c156430 tx=0x7f6880008040 comp rx=0 tx=0).stop 2023-07-04T18:52:23.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 msgr2=0x7f688c0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f687c009c10 tx=0x7f687c004750 comp rx=0 tx=0).stop 2023-07-04T18:52:23.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 shutdown_connections 2023-07-04T18:52:23.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f688c0d01c0 0x7f688c0b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f688c069e40 0x7f688c1f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 --2- 172.21.15.77:0/982511768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f688c0ae970 0x7f688c0b4300 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 >> 172.21.15.77:0/982511768 conn(0x7f688c06af50 msgr2=0x7f688c06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:23.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 shutdown_connections 2023-07-04T18:52:23.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.489+0000 7f68924b6700 1 -- 172.21.15.77:0/982511768 wait complete. 2023-07-04T18:52:23.505 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-07-04T18:52:23.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:23.629 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.628+0000 7f378ae8f700 1 Processor -- start 2023-07-04T18:52:23.630 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.628+0000 7f378ae8f700 1 -- start start 2023-07-04T18:52:23.632 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378ae8f700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 0x7f37840ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.632 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378ae8f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f37840ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.632 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378ae8f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3784071ad0 con 0x7f3784070f60 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378ae8f700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37840cb9f0 con 0x7f3784158c90 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378ae8f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840c9fd0 con 0x7f37840ce400 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378942c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3784070f60 0x7f37840713c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53078/0 (socket says 172.21.15.77:53078) 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f378942c700 1 -- 172.21.15.77:0/1722108442 learned_addr learned my addr 172.21.15.77:0/1722108442 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3788c2b700 1 --2- 172.21.15.77:0/1722108442 >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 0x7f37840ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3783fff700 1 --2- 172.21.15.77:0/1722108442 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f37840ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.633 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214448331 0 0) 0x7f3784071ad0 con 0x7f3784070f60 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37740035b0 con 0x7f3784070f60 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546287390 0 0) 0x7f37740035b0 con 0x7f3784070f60 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3783fff700 1 -- 172.21.15.77:0/1722108442 >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 msgr2=0x7f37840ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3783fff700 1 --2- 172.21.15.77:0/1722108442 >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 0x7f37840ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3783fff700 1 -- 172.21.15.77:0/1722108442 >> v1:172.21.15.77:6789/0 conn(0x7f3784070f60 legacy=0x7f37840713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.634 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3783fff700 1 -- 172.21.15.77:0/1722108442 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784071d50 con 0x7f37840ce400 2023-07-04T18:52:23.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.629+0000 7f3788c2b700 1 --2- 172.21.15.77:0/1722108442 >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 0x7f37840ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:23.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f3783fff700 1 --2- 172.21.15.77:0/1722108442 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f37840ce870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3770009840 tx=0x7f3770009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc896f31f3a41cbf server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:23.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 <== mon.2 v2:172.21.15.82:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0019f0 con 0x7f37840ce400 2023-07-04T18:52:23.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 <== mon.2 v2:172.21.15.82:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770004030 con 0x7f37840ce400 2023-07-04T18:52:23.635 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f37837fe700 1 -- 172.21.15.77:0/1722108442 <== mon.2 v2:172.21.15.82:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700043a0 con 0x7f37840ce400 2023-07-04T18:52:23.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 -- 172.21.15.77:0/1722108442 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 msgr2=0x7f37840ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 --2- 172.21.15.77:0/1722108442 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f37840ce870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3770009840 tx=0x7f3770009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:23.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 -- 172.21.15.77:0/1722108442 shutdown_connections 2023-07-04T18:52:23.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 --2- 172.21.15.77:0/1722108442 >> v2:172.21.15.82:3300/0 conn(0x7f3784158c90 0x7f37840ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.636 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 --2- 172.21.15.77:0/1722108442 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f37840ce870 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 -- 172.21.15.77:0/1722108442 >> 172.21.15.77:0/1722108442 conn(0x7f37840cf220 msgr2=0x7f37840cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:23.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 -- 172.21.15.77:0/1722108442 shutdown_connections 2023-07-04T18:52:23.637 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.630+0000 7f378ae8f700 1 -- 172.21.15.77:0/1722108442 wait complete. 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.632+0000 7f378ae8f700 1 Processor -- start 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.632+0000 7f378ae8f700 1 -- start start 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3784070f60 0x7f37841656f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f3784151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.640 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378409fe00 con 0x7f3784158c90 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840b2050 con 0x7f3784070f60 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840a0050 con 0x7f37840ce400 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45946/0 (socket says 172.21.15.77:45946) 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 -- 172.21.15.77:0/661586534 learned_addr learned my addr 172.21.15.77:0/661586534 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f3783fff700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f3784151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.641 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f3788c2b700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3784070f60 0x7f37841656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 -- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 msgr2=0x7f3784151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f3784151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 -- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3784070f60 msgr2=0x7f37841656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3784070f60 0x7f37841656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 -- 172.21.15.77:0/661586534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780094a0 con 0x7f3784158c90 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378942c700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f377c00cbe0 tx=0x7f377c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:23.642 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00ab80 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c00ad00 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37700094a0 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c007310 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.633+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37840b22a0 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.634+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37840b2610 con 0x7f3784158c90 2023-07-04T18:52:23.643 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.635+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f377c007470 con 0x7f3784158c90 2023-07-04T18:52:23.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.635+0000 7f37817fa700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 0x7f376806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:23.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.635+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f377c089420 con 0x7f3784158c90 2023-07-04T18:52:23.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.635+0000 7f3788c2b700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 0x7f376806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:23.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.636+0000 7f3788c2b700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 0x7f376806d190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3778009d00 tx=0x7f3778008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:23.644 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.639+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f377c054430 con 0x7f3784158c90 2023-07-04T18:52:23.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.870+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f378404e210 con 0x7f3784158c90 2023-07-04T18:52:23.872 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.870+0000 7f37817fa700 1 -- 172.21.15.77:0/661586534 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4668 (secure 0 0 0) 0x7f377c057a50 con 0x7f3784158c90 2023-07-04T18:52:23.872 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:23.873 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/1695626511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1695626511},{"type":"v1","addr":"172.21.15.82:6836","nonce":1695626511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3849345135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3849345135},{"type":"v1","addr":"172.21.15.77:6835","nonce":3849345135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3698064560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3698064560},{"type":"v1","addr":"172.21.15.77:6834","nonce":3698064560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:52:19.841356+0000","modified":"2023-07-04T18:52:23.480957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.82:6837/744114579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":744114579},{"type":"v1","addr":"172.21.15.82:6837","nonce":744114579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-07-04T18:52:23.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 msgr2=0x7f376806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 0x7f376806d190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3778009d00 tx=0x7f3778008040 comp rx=0 tx=0).stop 2023-07-04T18:52:23.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 msgr2=0x7f37840b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:23.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f377c00cbe0 tx=0x7f377c00a060 comp rx=0 tx=0).stop 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 shutdown_connections 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f376806ace0 0x7f376806d190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3784158c90 0x7f37840b27a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3784070f60 0x7f37841656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 --2- 172.21.15.77:0/661586534 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f37840ce400 0x7f3784151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 >> 172.21.15.77:0/661586534 conn(0x7f37840cf220 msgr2=0x7f37840cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:23.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 shutdown_connections 2023-07-04T18:52:23.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:23.872+0000 7f378ae8f700 1 -- 172.21.15.77:0/661586534 wait complete. 2023-07-04T18:52:23.876 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2023-07-04T18:52:23.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T18:52:24.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.006+0000 7f11c1a20700 1 Processor -- start 2023-07-04T18:52:24.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.006+0000 7f11c1a20700 1 -- start start 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11c1a20700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11bc0d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11c1a20700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc1610f0 con 0x7f11bc146270 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11c1a20700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc0b2790 con 0x7f11bc0af8f0 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11c1a20700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0b2a70 con 0x7f11bc0b1860 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11bb7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f11bc146270 0x7f11bc1609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53092/0 (socket says 172.21.15.77:53092) 2023-07-04T18:52:24.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11bb7fe700 1 -- 172.21.15.77:0/1288460108 learned_addr learned my addr 172.21.15.77:0/1288460108 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11baffd700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11bc0d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346610901 0 0) 0x7f11bc1610f0 con 0x7f11bc146270 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a40035b0 con 0x7f11bc146270 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469249174 0 0) 0x7f11bc0b2790 con 0x7f11bc0af8f0 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc1610f0 con 0x7f11bc0af8f0 2023-07-04T18:52:24.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 559779698 0 0) 0x7f11a40035b0 con 0x7f11bc146270 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 msgr2=0x7f11bc0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11bc0d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 >> v1:172.21.15.82:6789/0 conn(0x7f11bc0af8f0 legacy=0x7f11bc1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0b2d50 con 0x7f11bc146270 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f11b40019f0 con 0x7f11bc146270 2023-07-04T18:52:24.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.007+0000 7f11baffd700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11bc0d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 >> v1:172.21.15.77:6789/0 conn(0x7f11bc146270 legacy=0x7f11bc1609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11a4003bc0 0x7f11a4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a4004d90 0x7f11a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0b2a70 con 0x7f11a4004d90 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a400bac0 con 0x7f11a4003bc0 2023-07-04T18:52:24.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a400bc00 con 0x7f11bc0b1860 2023-07-04T18:52:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a4004d90 0x7f11a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11baffd700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11a4003bc0 0x7f11a4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 -- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 msgr2=0x7f11a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 -- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11a4003bc0 msgr2=0x7f11a4003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11a4003bc0 0x7f11a4003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 -- 172.21.15.77:0/1288460108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00094a0 con 0x7f11a4004d90 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0b1860 0x7f11a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11bb7fe700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a4004d90 0x7f11a4005200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f11b400a7c0 tx=0x7f11a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13ad724571c5f7 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4012600 con 0x7f11a4004d90 2023-07-04T18:52:24.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b4012780 con 0x7f11a4004d90 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.008+0000 7f11b9ffb700 1 -- 172.21.15.77:0/1288460108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4012b30 con 0x7f11a4004d90 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 -- 172.21.15.77:0/1288460108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a4004d90 msgr2=0x7f11a4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 --2- 172.21.15.77:0/1288460108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11a4004d90 0x7f11a4005200 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f11b400a7c0 tx=0x7f11a4005880 comp rx=0 tx=0).stop 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11bb7fe700 1 -- 172.21.15.77:0/1288460108 reap_dead start 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 -- 172.21.15.77:0/1288460108 shutdown_connections 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 -- 172.21.15.77:0/1288460108 >> 172.21.15.77:0/1288460108 conn(0x7f11bc071e90 msgr2=0x7f11bc09c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 -- 172.21.15.77:0/1288460108 shutdown_connections 2023-07-04T18:52:24.018 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.009+0000 7f11c1a20700 1 -- 172.21.15.77:0/1288460108 wait complete. 2023-07-04T18:52:24.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.010+0000 7f11c1a20700 1 Processor -- start 2023-07-04T18:52:24.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 -- start start 2023-07-04T18:52:24.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.020 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 0x7f11bc157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0c8860 0x7f11bc157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1f2160 con 0x7f11bc1553b0 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc155780 con 0x7f11bc0b83f0 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.011+0000 7f11c1a20700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0c8d00 con 0x7f11bc0c8860 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45962/0 (socket says 172.21.15.77:45962) 2023-07-04T18:52:24.021 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 -- 172.21.15.77:0/706460077 learned_addr learned my addr 172.21.15.77:0/706460077 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11bb7fe700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0c8860 0x7f11bc157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 0x7f11bc157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 -- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0c8860 msgr2=0x7f11bc157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0c8860 0x7f11bc157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 -- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 msgr2=0x7f11bc157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 0x7f11bc157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.022 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 -- 172.21.15.77:0/706460077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0c8fe0 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 0x7f11bc157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11baffd700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f11b0011ca0 tx=0x7f11b000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000fb30 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b000fc90 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc0c92c0 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0004020 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.012+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc1559d0 con 0x7f11bc1553b0 2023-07-04T18:52:24.023 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.013+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc1f22d0 con 0x7f11bc1553b0 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.013+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f11b0004180 con 0x7f11bc1553b0 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.014+0000 7f11a3fff700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 0x7f11a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.014+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f11b0089a30 con 0x7f11bc1553b0 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.014+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 0x7f11a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.014+0000 7f11ba7fc700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 0x7f11a806d190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f11bc09d000 tx=0x7f11ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.024 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.018+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11b0054ae0 con 0x7f11bc1553b0 2023-07-04T18:52:24.207 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.205+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11bc04e210 con 0x7f11bc1553b0 2023-07-04T18:52:24.208 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.206+0000 7f11a3fff700 1 -- 172.21.15.77:0/706460077 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18315 (secure 0 0 0) 0x7f11b0022090 con 0x7f11bc1553b0 2023-07-04T18:52:24.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:24.209 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","created":"2023-07-04T18:50:57.548255+0000","modified":"2023-07-04T18:52:21.462954+0000","last_up_change":"2023-07-04T18:51:09.134847+0000","last_in_change":"2023-07-04T18:51:00.022059+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-04T18:51:09.218926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-04T18:51:15.974768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-07-04T18:52:17.964674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-07-04T18:52:18.802651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ece9a66-e00e-486d-860c-1ee57a7de146","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6809","nonce":4272426713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6811","nonce":4272426713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6815","nonce":4272426713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":4272426713},{"type":"v1","addr":"172.21.15.77:6813","nonce":4272426713}]},"public_addr":"172.21.15.77:6809/4272426713","cluster_addr":"172.21.15.77:6811/4272426713","heartbeat_back_addr":"172.21.15.77:6815/4272426713","heartbeat_front_addr":"172.21.15.77:6813/4272426713","state":["exists","up"]},{"osd":1,"uuid":"c2fddd95-3531-4fc8-8749-f8e4d50b7b1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6801","nonce":1069379231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6803","nonce":1069379231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6807","nonce":1069379231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":1069379231},{"type":"v1","addr":"172.21.15.77:6805","nonce":1069379231}]},"public_addr":"172.21.15.77:6801/1069379231","cluster_addr":"172.21.15.77:6803/1069379231","heartbeat_back_addr":"172.21.15.77:6807/1069379231","heartbeat_front_addr":"172.21.15.77:6805/1069379231","state":["exists","up"]},{"osd":2,"uuid":"9f71759a-fb46-4929-a09d-1a36386ac3bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6825","nonce":3740385611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6827","nonce":3740385611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6831","nonce":3740385611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3740385611},{"type":"v1","addr":"172.21.15.77:6829","nonce":3740385611}]},"public_addr":"172.21.15.77:6825/3740385611","cluster_addr":"172.21.15.77:6827/3740385611","heartbeat_back_addr":"172.21.15.77:6831/3740385611","heartbeat_front_addr":"172.21.15.77:6829/3740385611","state":["exists","up"]},{"osd":3,"uuid":"fe8befe0-56b7-4f0c-bc08-910c2ec4637a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6817","nonce":1589412867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6819","nonce":1589412867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6823","nonce":1589412867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1589412867},{"type":"v1","addr":"172.21.15.77:6821","nonce":1589412867}]},"public_addr":"172.21.15.77:6817/1589412867","cluster_addr":"172.21.15.77:6819/1589412867","heartbeat_back_addr":"172.21.15.77:6823/1589412867","heartbeat_front_addr":"172.21.15.77:6821/1589412867","state":["exists","up"]},{"osd":4,"uuid":"8dada326-1025-469e-93b0-0e7e4aa3cdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6801","nonce":1338358939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6803","nonce":1338358939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6807","nonce":1338358939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1338358939},{"type":"v1","addr":"172.21.15.82:6805","nonce":1338358939}]},"public_addr":"172.21.15.82:6801/1338358939","cluster_addr":"172.21.15.82:6803/1338358939","heartbeat_back_addr":"172.21.15.82:6807/1338358939","heartbeat_front_addr":"172.21.15.82:6805/1338358939","state":["exists","up"]},{"osd":5,"uuid":"53f40546-3ced-4dcd-8ca1-b3ca6a0a576d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6817","nonce":2297621363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6819","nonce":2297621363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6823","nonce":2297621363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2297621363},{"type":"v1","addr":"172.21.15.82:6821","nonce":2297621363}]},"public_addr":"172.21.15.82:6817/2297621363","cluster_addr":"172.21.15.82:6819/2297621363","heartbeat_back_addr":"172.21.15.82:6823/2297621363","heartbeat_front_addr":"172.21.15.82:6821/2297621363","state":["exists","up"]},{"osd":6,"uuid":"b25b7f3b-c465-4ecc-9cef-9071c5c5b9b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6809","nonce":2390446703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6811","nonce":2390446703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6815","nonce":2390446703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":2390446703},{"type":"v1","addr":"172.21.15.82:6813","nonce":2390446703}]},"public_addr":"172.21.15.82:6809/2390446703","cluster_addr":"172.21.15.82:6811/2390446703","heartbeat_back_addr":"172.21.15.82:6815/2390446703","heartbeat_front_addr":"172.21.15.82:6813/2390446703","state":["exists","up"]},{"osd":7,"uuid":"453bc1b4-29d7-485e-9771-38679a8bc1ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6825","nonce":3741415560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6827","nonce":3741415560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6831","nonce":3741415560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3741415560},{"type":"v1","addr":"172.21.15.82:6829","nonce":3741415560}]},"public_addr":"172.21.15.82:6825/3741415560","cluster_addr":"172.21.15.82:6827/3741415560","heartbeat_back_addr":"172.21.15.82:6831/3741415560","heartbeat_front_addr":"172.21.15.82:6829/3741415560","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.251853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.178201+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.222584+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.252311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.290188+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.177659+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.219491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-04T18:51:05.278361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-04T18:52:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 msgr2=0x7f11a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.211 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 0x7f11a806d190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f11bc09d000 tx=0x7f11ac009040 comp rx=0 tx=0).stop 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 msgr2=0x7f11bc0c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f11b0011ca0 tx=0x7f11b000f520 comp rx=0 tx=0).stop 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 shutdown_connections 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f11a806ace0 0x7f11a806d190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc1553b0 0x7f11bc0c8320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.212 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f11bc0c8860 0x7f11bc157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 --2- 172.21.15.77:0/706460077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11bc0b83f0 0x7f11bc157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 >> 172.21.15.77:0/706460077 conn(0x7f11bc071e90 msgr2=0x7f11bc0d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 shutdown_connections 2023-07-04T18:52:24.213 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.209+0000 7f11c1a20700 1 -- 172.21.15.77:0/706460077 wait complete. 2023-07-04T18:52:24.224 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-04T18:52:24.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:52:24.346 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.345+0000 7f5d498c4700 1 Processor -- start 2023-07-04T18:52:24.347 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.345+0000 7f5d498c4700 1 -- start start 2023-07-04T18:52:24.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d498c4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d440b1860 0x7f5d440d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d498c4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d440af8f0 0x7f5d440d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d498c4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d440b2790 con 0x7f5d44146270 2023-07-04T18:52:24.349 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d498c4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d440b2a70 con 0x7f5d440b1860 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d498c4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440b2d50 con 0x7f5d440af8f0 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d440b1860 0x7f5d440d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d437fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5d44146270 0x7f5d441602d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53112/0 (socket says 172.21.15.77:53112) 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5d440b1860 0x7f5d440d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45970/0 (socket says 172.21.15.77:45970) 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d437fe700 1 -- 172.21.15.77:0/2698593701 learned_addr learned my addr 172.21.15.77:0/2698593701 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d427fc700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d440af8f0 0x7f5d440d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.350 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 -- 172.21.15.77:0/2698593701 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d440af8f0 msgr2=0x7f5d440d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d440af8f0 0x7f5d440d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 -- 172.21.15.77:0/2698593701 >> v1:172.21.15.77:6789/0 conn(0x7f5d44146270 legacy=0x7f5d441602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 -- 172.21.15.77:0/2698593701 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d440b3030 con 0x7f5d440b1860 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d42ffd700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440b1860 0x7f5d440d2cb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009840 tx=0x7f5d38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a56648872805d4 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d41ffb700 1 -- 172.21.15.77:0/2698593701 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38011830 con 0x7f5d440b1860 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d41ffb700 1 -- 172.21.15.77:0/2698593701 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d38011990 con 0x7f5d440b1860 2023-07-04T18:52:24.351 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.346+0000 7f5d41ffb700 1 -- 172.21.15.77:0/2698593701 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38011d20 con 0x7f5d440b1860 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 -- 172.21.15.77:0/2698593701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440b1860 msgr2=0x7f5d440d2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440b1860 0x7f5d440d2cb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009840 tx=0x7f5d38009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 -- 172.21.15.77:0/2698593701 shutdown_connections 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440b1860 0x7f5d440d2cb0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 --2- 172.21.15.77:0/2698593701 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d440af8f0 0x7f5d440d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 -- 172.21.15.77:0/2698593701 >> 172.21.15.77:0/2698593701 conn(0x7f5d44071e90 msgr2=0x7f5d4409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.352 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 -- 172.21.15.77:0/2698593701 shutdown_connections 2023-07-04T18:52:24.353 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.347+0000 7f5d498c4700 1 -- 172.21.15.77:0/2698593701 wait complete. 2023-07-04T18:52:24.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.348+0000 7f5d498c4700 1 Processor -- start 2023-07-04T18:52:24.356 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.349+0000 7f5d498c4700 1 -- start start 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.349+0000 7f5d498c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d440b1860 0x7f5d440c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d44146270 0x7f5d441f12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440c9e60 con 0x7f5d440af8f0 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440c7bd0 con 0x7f5d440b1860 2023-07-04T18:52:24.357 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d440ca070 con 0x7f5d44146270 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45976/0 (socket says 172.21.15.77:45976) 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 -- 172.21.15.77:0/3829066383 learned_addr learned my addr 172.21.15.77:0/3829066383 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 -- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d44146270 msgr2=0x7f5d441f12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d44146270 0x7f5d441f12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 -- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d440b1860 msgr2=0x7f5d440c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d440b1860 0x7f5d440c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.358 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 -- 172.21.15.77:0/3829066383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d380094a0 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d42ffd700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009b90 tx=0x7f5d38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d380038a0 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3801f070 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38003c00 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d440ca350 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.350+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d440c7810 con 0x7f5d440af8f0 2023-07-04T18:52:24.359 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.351+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d440c79a0 con 0x7f5d440af8f0 2023-07-04T18:52:24.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.352+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d3801a430 con 0x7f5d440af8f0 2023-07-04T18:52:24.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.352+0000 7f5d2bfff700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 0x7f5d3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.352+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5d38021070 con 0x7f5d440af8f0 2023-07-04T18:52:24.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.352+0000 7f5d427fc700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 0x7f5d3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.360 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.353+0000 7f5d427fc700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 0x7f5d3006d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d44159df0 tx=0x7f5d34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.361 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.356+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d38056020 con 0x7f5d440af8f0 2023-07-04T18:52:24.541 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.540+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d440b3310 con 0x7f5d3006ace0 2023-07-04T18:52:24.554 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.549+0000 7f5d2bfff700 1 -- 172.21.15.77:0/3829066383 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299591 (secure 0 0 0) 0x7f5d340133f0 con 0x7f5d3006ace0 2023-07-04T18:52:24.560 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:24.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 msgr2=0x7f5d3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 0x7f5d3006d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d44159df0 tx=0x7f5d34008040 comp rx=0 tx=0).stop 2023-07-04T18:52:24.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 msgr2=0x7f5d440c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.574 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009b90 tx=0x7f5d38004750 comp rx=0 tx=0).stop 2023-07-04T18:52:24.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 shutdown_connections 2023-07-04T18:52:24.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5d3006ace0 0x7f5d3006d190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.575 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d440af8f0 0x7f5d440c72d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5d44146270 0x7f5d441f12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 --2- 172.21.15.77:0/3829066383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5d440b1860 0x7f5d440c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 >> 172.21.15.77:0/3829066383 conn(0x7f5d44071e90 msgr2=0x7f5d440afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 shutdown_connections 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.552+0000 7f5d498c4700 1 -- 172.21.15.77:0/3829066383 wait complete. 2023-07-04T18:52:24.576 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:52:24.577 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2023-07-04T18:52:23.191650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219760,"kb_used_data":4144,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754960,"statfs":{"total":773094113280,"available":772869079040,"internally_reserved":0,"allocated":4243456,"data_stored":2408228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.753439"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440959+0000","last_change":"2023-07-04T18:52:18.440959+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440959+0000","last_undegraded":"2023-07-04T18:52:18.440959+0000","last_fullsized":"2023-07-04T18:52:18.440959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443525+0000","last_change":"2023-07-04T18:52:18.443525+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443525+0000","last_undegraded":"2023-07-04T18:52:18.443525+0000","last_fullsized":"2023-07-04T18:52:18.443525+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446682+0000","last_change":"2023-07-04T18:52:18.446682+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446682+0000","last_undegraded":"2023-07-04T18:52:18.446682+0000","last_fullsized":"2023-07-04T18:52:18.446682+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440344+0000","last_change":"2023-07-04T18:52:18.440344+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440344+0000","last_undegraded":"2023-07-04T18:52:18.440344+0000","last_fullsized":"2023-07-04T18:52:18.440344+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443295+0000","last_change":"2023-07-04T18:52:18.443295+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443295+0000","last_undegraded":"2023-07-04T18:52:18.443295+0000","last_fullsized":"2023-07-04T18:52:18.443295+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440192+0000","last_change":"2023-07-04T18:52:18.440192+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440192+0000","last_undegraded":"2023-07-04T18:52:18.440192+0000","last_fullsized":"2023-07-04T18:52:18.440192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446925+0000","last_change":"2023-07-04T18:52:18.446925+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446925+0000","last_undegraded":"2023-07-04T18:52:18.446925+0000","last_fullsized":"2023-07-04T18:52:18.446925+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441578+0000","last_change":"2023-07-04T18:52:18.441578+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441578+0000","last_undegraded":"2023-07-04T18:52:18.441578+0000","last_fullsized":"2023-07-04T18:52:18.441578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442379+0000","last_change":"2023-07-04T18:52:18.442379+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442379+0000","last_undegraded":"2023-07-04T18:52:18.442379+0000","last_fullsized":"2023-07-04T18:52:18.442379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443340+0000","last_change":"2023-07-04T18:52:18.443340+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443340+0000","last_undegraded":"2023-07-04T18:52:18.443340+0000","last_fullsized":"2023-07-04T18:52:18.443340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443392+0000","last_change":"2023-07-04T18:52:18.443392+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443392+0000","last_undegraded":"2023-07-04T18:52:18.443392+0000","last_fullsized":"2023-07-04T18:52:18.443392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442676+0000","last_change":"2023-07-04T18:52:18.442676+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442676+0000","last_undegraded":"2023-07-04T18:52:18.442676+0000","last_fullsized":"2023-07-04T18:52:18.442676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442377+0000","last_change":"2023-07-04T18:52:18.442377+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442377+0000","last_undegraded":"2023-07-04T18:52:18.442377+0000","last_fullsized":"2023-07-04T18:52:18.442377+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446661+0000","last_change":"2023-07-04T18:52:18.446661+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446661+0000","last_undegraded":"2023-07-04T18:52:18.446661+0000","last_fullsized":"2023-07-04T18:52:18.446661+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440885+0000","last_change":"2023-07-04T18:52:18.440885+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440885+0000","last_undegraded":"2023-07-04T18:52:18.440885+0000","last_fullsized":"2023-07-04T18:52:18.440885+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441267+0000","last_change":"2023-07-04T18:52:18.441267+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441267+0000","last_undegraded":"2023-07-04T18:52:18.441267+0000","last_fullsized":"2023-07-04T18:52:18.441267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443915+0000","last_change":"2023-07-04T18:52:18.443915+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443915+0000","last_undegraded":"2023-07-04T18:52:18.443915+0000","last_fullsized":"2023-07-04T18:52:18.443915+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441184+0000","last_change":"2023-07-04T18:52:18.441184+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441184+0000","last_undegraded":"2023-07-04T18:52:18.441184+0000","last_fullsized":"2023-07-04T18:52:18.441184+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444746+0000","last_change":"2023-07-04T18:52:18.444746+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444746+0000","last_undegraded":"2023-07-04T18:52:18.444746+0000","last_fullsized":"2023-07-04T18:52:18.444746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446765+0000","last_change":"2023-07-04T18:52:18.446765+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446765+0000","last_undegraded":"2023-07-04T18:52:18.446765+0000","last_fullsized":"2023-07-04T18:52:18.446765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444743+0000","last_change":"2023-07-04T18:52:18.444743+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444743+0000","last_undegraded":"2023-07-04T18:52:18.444743+0000","last_fullsized":"2023-07-04T18:52:18.444743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.445082+0000","last_change":"2023-07-04T18:52:18.445082+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.445082+0000","last_undegraded":"2023-07-04T18:52:18.445082+0000","last_fullsized":"2023-07-04T18:52:18.445082+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440101+0000","last_change":"2023-07-04T18:52:18.440101+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440101+0000","last_undegraded":"2023-07-04T18:52:18.440101+0000","last_fullsized":"2023-07-04T18:52:18.440101+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444685+0000","last_change":"2023-07-04T18:52:18.444685+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444685+0000","last_undegraded":"2023-07-04T18:52:18.444685+0000","last_fullsized":"2023-07-04T18:52:18.444685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443233+0000","last_change":"2023-07-04T18:52:18.443233+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443233+0000","last_undegraded":"2023-07-04T18:52:18.443233+0000","last_fullsized":"2023-07-04T18:52:18.443233+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.447247+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:52:18.447247+0000","last_peered":"2023-07-04T18:52:18.447247+0000","last_clean":"2023-07-04T18:52:18.447247+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:52:18.447247+0000","last_undegraded":"2023-07-04T18:52:18.447247+0000","last_fullsized":"2023-07-04T18:52:18.447247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:13:43.391691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443161+0000","last_change":"2023-07-04T18:52:18.443161+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443161+0000","last_undegraded":"2023-07-04T18:52:18.443161+0000","last_fullsized":"2023-07-04T18:52:18.443161+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.444427+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:52:18.444427+0000","last_peered":"2023-07-04T18:52:18.444427+0000","last_clean":"2023-07-04T18:52:18.444427+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:52:18.444427+0000","last_undegraded":"2023-07-04T18:52:18.444427+0000","last_fullsized":"2023-07-04T18:52:18.444427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:46:17.210983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.445335+0000","last_change":"2023-07-04T18:52:18.445335+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.445335+0000","last_undegraded":"2023-07-04T18:52:18.445335+0000","last_fullsized":"2023-07-04T18:52:18.445335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.445642+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:52:18.445642+0000","last_peered":"2023-07-04T18:52:18.445642+0000","last_clean":"2023-07-04T18:52:18.445642+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:52:18.445642+0000","last_undegraded":"2023-07-04T18:52:18.445642+0000","last_fullsized":"2023-07-04T18:52:18.445642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:13:34.538196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441710+0000","last_change":"2023-07-04T18:52:18.441710+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441710+0000","last_undegraded":"2023-07-04T18:52:18.441710+0000","last_fullsized":"2023-07-04T18:52:18.441710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.444439+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:52:18.444439+0000","last_peered":"2023-07-04T18:52:18.444439+0000","last_clean":"2023-07-04T18:52:18.444439+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:52:18.444439+0000","last_undegraded":"2023-07-04T18:52:18.444439+0000","last_fullsized":"2023-07-04T18:52:18.444439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:27:24.306913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":191,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442080+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:52:18.442080+0000","last_peered":"2023-07-04T18:52:18.442080+0000","last_clean":"2023-07-04T18:52:18.442080+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:52:18.442080+0000","last_undegraded":"2023-07-04T18:52:18.442080+0000","last_fullsized":"2023-07-04T18:52:18.442080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:43:15.786945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443376+0000","last_change":"2023-07-04T18:52:18.443376+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443376+0000","last_undegraded":"2023-07-04T18:52:18.443376+0000","last_fullsized":"2023-07-04T18:52:18.443376+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442226+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:52:18.442226+0000","last_peered":"2023-07-04T18:52:18.442226+0000","last_clean":"2023-07-04T18:52:18.442226+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:52:18.442226+0000","last_undegraded":"2023-07-04T18:52:18.442226+0000","last_fullsized":"2023-07-04T18:52:18.442226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:13:52.861879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440692+0000","last_change":"2023-07-04T18:52:18.440692+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440692+0000","last_undegraded":"2023-07-04T18:52:18.440692+0000","last_fullsized":"2023-07-04T18:52:18.440692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442502+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:52:18.442502+0000","last_peered":"2023-07-04T18:52:18.442502+0000","last_clean":"2023-07-04T18:52:18.442502+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:52:18.442502+0000","last_undegraded":"2023-07-04T18:52:18.442502+0000","last_fullsized":"2023-07-04T18:52:18.442502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:45:33.859559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440413+0000","last_change":"2023-07-04T18:52:18.440413+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440413+0000","last_undegraded":"2023-07-04T18:52:18.440413+0000","last_fullsized":"2023-07-04T18:52:18.440413+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.447259+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:52:18.447259+0000","last_peered":"2023-07-04T18:52:18.447259+0000","last_clean":"2023-07-04T18:52:18.447259+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:52:18.447259+0000","last_undegraded":"2023-07-04T18:52:18.447259+0000","last_fullsized":"2023-07-04T18:52:18.447259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:53:11.058829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442368+0000","last_change":"2023-07-04T18:52:18.442368+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442368+0000","last_undegraded":"2023-07-04T18:52:18.442368+0000","last_fullsized":"2023-07-04T18:52:18.442368+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.439560+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:52:18.439560+0000","last_peered":"2023-07-04T18:52:18.439560+0000","last_clean":"2023-07-04T18:52:18.439560+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:52:18.439560+0000","last_undegraded":"2023-07-04T18:52:18.439560+0000","last_fullsized":"2023-07-04T18:52:18.439560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:39:26.295849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":978944,"data_stored":745347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55500000000000005}]},{"osd":1,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71899999999999997}]},{"osd":6,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.56000000000000005}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.621},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.504}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":372736,"data_stored":148809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.34000000000000002}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":3.0600000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.434}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.42499999999999999}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.498},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.46500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":978944,"data_stored":745347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.217},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.29199999999999998}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.311},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.32500000000000001}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.7719999999999998,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":3.7109999999999999,"5min":3.7109999999999999,"15min":3.7109999999999999},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.44700000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.217}]},{"osd":1,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28599999999999998}]},{"osd":3,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.313},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.30099999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51500000000000001}]},{"osd":6,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34899999999999998}]},{"osd":1,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.317}]},{"osd":2,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.38},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.27900000000000003}]},{"osd":4,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.503},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.46200000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.54900000000000004}]},{"osd":6,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.47099999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.499}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.375},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59099999999999997}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61499999999999999}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":372736,"data_stored":148809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.50700000000000001}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.95799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:52:24.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:52:24.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.733+0000 7fd0ab655700 1 Processor -- start 2023-07-04T18:52:24.735 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.733+0000 7fd0ab655700 1 -- start start 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0ab655700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd0a40cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0ab655700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a4159250 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0ab655700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a4159530 con 0x7fd0a406ba80 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0ab655700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a4159ed0 con 0x7fd0a40cab70 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0a9bf2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd0a40cc9f0 0x7fd0a40cce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53126/0 (socket says 172.21.15.77:53126) 2023-07-04T18:52:24.738 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/1294477752 learned_addr learned my addr 172.21.15.77:0/1294477752 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd0a40cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322945428 0 0) 0x7fd0a4159250 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0900035b0 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.734+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199277852 0 0) 0x7fd0a4159530 con 0x7fd0a406ba80 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a4159250 con 0x7fd0a406ba80 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648071467 0 0) 0x7fd0900035b0 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.739 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 msgr2=0x7fd0a40cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd0a40cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 >> v1:172.21.15.82:6789/0 conn(0x7fd0a406ba80 legacy=0x7fd0a406be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4159810 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7fd09c001a40 con 0x7fd0a40cc9f0 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 >> v1:172.21.15.77:6789/0 conn(0x7fd0a40cc9f0 legacy=0x7fd0a40cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.740 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd090003bc0 0x7fd090003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd090004850 0x7fd090004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.741 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd090005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.743 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a4159ed0 con 0x7fd090004850 2023-07-04T18:52:24.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09000bb20 con 0x7fd090003bc0 2023-07-04T18:52:24.744 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09000bce0 con 0x7fd0a40cab70 2023-07-04T18:52:24.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd090004850 0x7fd090004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a93f1700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd090003bc0 0x7fd090003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a9bf2700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd090005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.745 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 -- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 msgr2=0x7fd090005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd0a40cab70 0x7fd090005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 -- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd090003bc0 msgr2=0x7fd090003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:24.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd090003bc0 0x7fd090003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.746 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 -- 172.21.15.77:0/1294477752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900035b0 con 0x7fd090004850 2023-07-04T18:52:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd090004850 0x7fd090004cc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000ab60 tx=0x7fd090005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2b6780c4835e46 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000f9d0 con 0x7fd090004850 2023-07-04T18:52:24.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a000fb30 con 0x7fd090004850 2023-07-04T18:52:24.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.735+0000 7fd09bfff700 1 -- 172.21.15.77:0/1294477752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000fea0 con 0x7fd090004850 2023-07-04T18:52:24.748 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 -- 172.21.15.77:0/1294477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd090004850 msgr2=0x7fd090004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.749 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 --2- 172.21.15.77:0/1294477752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd090004850 0x7fd090004cc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000ab60 tx=0x7fd090005880 comp rx=0 tx=0).stop 2023-07-04T18:52:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/1294477752 reap_dead start 2023-07-04T18:52:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 -- 172.21.15.77:0/1294477752 shutdown_connections 2023-07-04T18:52:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 -- 172.21.15.77:0/1294477752 >> 172.21.15.77:0/1294477752 conn(0x7fd0a406aaa0 msgr2=0x7fd0a40b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 -- 172.21.15.77:0/1294477752 shutdown_connections 2023-07-04T18:52:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.736+0000 7fd0ab655700 1 -- 172.21.15.77:0/1294477752 wait complete. 2023-07-04T18:52:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.737+0000 7fd0ab655700 1 Processor -- start 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.737+0000 7fd0ab655700 1 -- start start 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd090004850 0x7fd0a414bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0a414ccb0 0x7fd0a4155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a414ca40 con 0x7fd0a414e4f0 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a414c510 con 0x7fd0a414ccb0 2023-07-04T18:52:24.753 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0ab655700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a414eed0 con 0x7fd090004850 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45998/0 (socket says 172.21.15.77:45998) 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/2830832483 learned_addr learned my addr 172.21.15.77:0/2830832483 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a8bf0700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0a414ccb0 0x7fd0a4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a93f1700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd090004850 0x7fd0a414bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd090004850 msgr2=0x7fd0a414bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd090004850 0x7fd0a414bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0a414ccb0 msgr2=0x7fd0a4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0a414ccb0 0x7fd0a4155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 -- 172.21.15.77:0/2830832483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a414d120 con 0x7fd0a414e4f0 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.738+0000 7fd0a9bf2700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40c5da0 tx=0x7fd09c00a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.739+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0014a0 con 0x7fd0a414e4f0 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.739+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c015410 con 0x7fd0a414e4f0 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.739+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0157c0 con 0x7fd0a414e4f0 2023-07-04T18:52:24.755 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.739+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a414d400 con 0x7fd0a414e4f0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.739+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a414d9c0 con 0x7fd0a414e4f0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.740+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a404e210 con 0x7fd0a414e4f0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.740+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd09c007d50 con 0x7fd0a414e4f0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.741+0000 7fd099ffb700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 0x7fd08806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.741+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fd09c08be10 con 0x7fd0a414e4f0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.741+0000 7fd0a93f1700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 0x7fd08806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:24.756 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.742+0000 7fd0a93f1700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 0x7fd08806d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd094007fd0 tx=0x7fd094009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:24.757 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.745+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd09c056f50 con 0x7fd0a414e4f0 2023-07-04T18:52:24.928 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.926+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0a4159af0 con 0x7fd08806ace0 2023-07-04T18:52:24.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.934+0000 7fd099ffb700 1 -- 172.21.15.77:0/2830832483 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+299591 (secure 0 0 0) 0x7fd0940051d0 con 0x7fd08806ace0 2023-07-04T18:52:24.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 msgr2=0x7fd08806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 0x7fd08806d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd094007fd0 tx=0x7fd094009040 comp rx=0 tx=0).stop 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 msgr2=0x7fd0a414e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40c5da0 tx=0x7fd09c00a7f0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 shutdown_connections 2023-07-04T18:52:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fd08806ace0 0x7fd08806d190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0a414e4f0 0x7fd0a414e990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fd090004850 0x7fd0a414bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 --2- 172.21.15.77:0/2830832483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0a414ccb0 0x7fd0a4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 >> 172.21.15.77:0/2830832483 conn(0x7fd0a406aaa0 msgr2=0x7fd0a4070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 shutdown_connections 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:24.938+0000 7fd0ab655700 1 -- 172.21.15.77:0/2830832483 wait complete. 2023-07-04T18:52:24.942 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:52:24.951 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2023-07-04T18:52:23.191650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219760,"kb_used_data":4144,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754960,"statfs":{"total":773094113280,"available":772869079040,"internally_reserved":0,"allocated":4243456,"data_stored":2408228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.753439"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440959+0000","last_change":"2023-07-04T18:52:18.440959+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440959+0000","last_undegraded":"2023-07-04T18:52:18.440959+0000","last_fullsized":"2023-07-04T18:52:18.440959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443525+0000","last_change":"2023-07-04T18:52:18.443525+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443525+0000","last_undegraded":"2023-07-04T18:52:18.443525+0000","last_fullsized":"2023-07-04T18:52:18.443525+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446682+0000","last_change":"2023-07-04T18:52:18.446682+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446682+0000","last_undegraded":"2023-07-04T18:52:18.446682+0000","last_fullsized":"2023-07-04T18:52:18.446682+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440344+0000","last_change":"2023-07-04T18:52:18.440344+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440344+0000","last_undegraded":"2023-07-04T18:52:18.440344+0000","last_fullsized":"2023-07-04T18:52:18.440344+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443295+0000","last_change":"2023-07-04T18:52:18.443295+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443295+0000","last_undegraded":"2023-07-04T18:52:18.443295+0000","last_fullsized":"2023-07-04T18:52:18.443295+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440192+0000","last_change":"2023-07-04T18:52:18.440192+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440192+0000","last_undegraded":"2023-07-04T18:52:18.440192+0000","last_fullsized":"2023-07-04T18:52:18.440192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446925+0000","last_change":"2023-07-04T18:52:18.446925+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446925+0000","last_undegraded":"2023-07-04T18:52:18.446925+0000","last_fullsized":"2023-07-04T18:52:18.446925+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441578+0000","last_change":"2023-07-04T18:52:18.441578+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441578+0000","last_undegraded":"2023-07-04T18:52:18.441578+0000","last_fullsized":"2023-07-04T18:52:18.441578+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442379+0000","last_change":"2023-07-04T18:52:18.442379+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442379+0000","last_undegraded":"2023-07-04T18:52:18.442379+0000","last_fullsized":"2023-07-04T18:52:18.442379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443340+0000","last_change":"2023-07-04T18:52:18.443340+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443340+0000","last_undegraded":"2023-07-04T18:52:18.443340+0000","last_fullsized":"2023-07-04T18:52:18.443340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443392+0000","last_change":"2023-07-04T18:52:18.443392+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443392+0000","last_undegraded":"2023-07-04T18:52:18.443392+0000","last_fullsized":"2023-07-04T18:52:18.443392+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442676+0000","last_change":"2023-07-04T18:52:18.442676+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442676+0000","last_undegraded":"2023-07-04T18:52:18.442676+0000","last_fullsized":"2023-07-04T18:52:18.442676+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442377+0000","last_change":"2023-07-04T18:52:18.442377+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442377+0000","last_undegraded":"2023-07-04T18:52:18.442377+0000","last_fullsized":"2023-07-04T18:52:18.442377+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446661+0000","last_change":"2023-07-04T18:52:18.446661+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446661+0000","last_undegraded":"2023-07-04T18:52:18.446661+0000","last_fullsized":"2023-07-04T18:52:18.446661+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440885+0000","last_change":"2023-07-04T18:52:18.440885+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440885+0000","last_undegraded":"2023-07-04T18:52:18.440885+0000","last_fullsized":"2023-07-04T18:52:18.440885+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441267+0000","last_change":"2023-07-04T18:52:18.441267+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441267+0000","last_undegraded":"2023-07-04T18:52:18.441267+0000","last_fullsized":"2023-07-04T18:52:18.441267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443915+0000","last_change":"2023-07-04T18:52:18.443915+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443915+0000","last_undegraded":"2023-07-04T18:52:18.443915+0000","last_fullsized":"2023-07-04T18:52:18.443915+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441184+0000","last_change":"2023-07-04T18:52:18.441184+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441184+0000","last_undegraded":"2023-07-04T18:52:18.441184+0000","last_fullsized":"2023-07-04T18:52:18.441184+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444746+0000","last_change":"2023-07-04T18:52:18.444746+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444746+0000","last_undegraded":"2023-07-04T18:52:18.444746+0000","last_fullsized":"2023-07-04T18:52:18.444746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.446765+0000","last_change":"2023-07-04T18:52:18.446765+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.446765+0000","last_undegraded":"2023-07-04T18:52:18.446765+0000","last_fullsized":"2023-07-04T18:52:18.446765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444743+0000","last_change":"2023-07-04T18:52:18.444743+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444743+0000","last_undegraded":"2023-07-04T18:52:18.444743+0000","last_fullsized":"2023-07-04T18:52:18.444743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.445082+0000","last_change":"2023-07-04T18:52:18.445082+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.445082+0000","last_undegraded":"2023-07-04T18:52:18.445082+0000","last_fullsized":"2023-07-04T18:52:18.445082+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440101+0000","last_change":"2023-07-04T18:52:18.440101+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440101+0000","last_undegraded":"2023-07-04T18:52:18.440101+0000","last_fullsized":"2023-07-04T18:52:18.440101+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.444685+0000","last_change":"2023-07-04T18:52:18.444685+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.444685+0000","last_undegraded":"2023-07-04T18:52:18.444685+0000","last_fullsized":"2023-07-04T18:52:18.444685+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443233+0000","last_change":"2023-07-04T18:52:18.443233+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443233+0000","last_undegraded":"2023-07-04T18:52:18.443233+0000","last_fullsized":"2023-07-04T18:52:18.443233+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.447247+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:52:18.447247+0000","last_peered":"2023-07-04T18:52:18.447247+0000","last_clean":"2023-07-04T18:52:18.447247+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:52:18.447247+0000","last_undegraded":"2023-07-04T18:52:18.447247+0000","last_fullsized":"2023-07-04T18:52:18.447247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:13:43.391691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443161+0000","last_change":"2023-07-04T18:52:18.443161+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443161+0000","last_undegraded":"2023-07-04T18:52:18.443161+0000","last_fullsized":"2023-07-04T18:52:18.443161+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.444427+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:52:18.444427+0000","last_peered":"2023-07-04T18:52:18.444427+0000","last_clean":"2023-07-04T18:52:18.444427+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:52:18.444427+0000","last_undegraded":"2023-07-04T18:52:18.444427+0000","last_fullsized":"2023-07-04T18:52:18.444427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:46:17.210983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.445335+0000","last_change":"2023-07-04T18:52:18.445335+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.445335+0000","last_undegraded":"2023-07-04T18:52:18.445335+0000","last_fullsized":"2023-07-04T18:52:18.445335+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.445642+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:52:18.445642+0000","last_peered":"2023-07-04T18:52:18.445642+0000","last_clean":"2023-07-04T18:52:18.445642+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:52:18.445642+0000","last_undegraded":"2023-07-04T18:52:18.445642+0000","last_fullsized":"2023-07-04T18:52:18.445642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:13:34.538196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.441710+0000","last_change":"2023-07-04T18:52:18.441710+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.441710+0000","last_undegraded":"2023-07-04T18:52:18.441710+0000","last_fullsized":"2023-07-04T18:52:18.441710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.444439+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:52:18.444439+0000","last_peered":"2023-07-04T18:52:18.444439+0000","last_clean":"2023-07-04T18:52:18.444439+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:52:18.444439+0000","last_undegraded":"2023-07-04T18:52:18.444439+0000","last_fullsized":"2023-07-04T18:52:18.444439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:27:24.306913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":191,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442080+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:52:18.442080+0000","last_peered":"2023-07-04T18:52:18.442080+0000","last_clean":"2023-07-04T18:52:18.442080+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:52:18.442080+0000","last_undegraded":"2023-07-04T18:52:18.442080+0000","last_fullsized":"2023-07-04T18:52:18.442080+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:43:15.786945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.443376+0000","last_change":"2023-07-04T18:52:18.443376+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.443376+0000","last_undegraded":"2023-07-04T18:52:18.443376+0000","last_fullsized":"2023-07-04T18:52:18.443376+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442226+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:52:18.442226+0000","last_peered":"2023-07-04T18:52:18.442226+0000","last_clean":"2023-07-04T18:52:18.442226+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:52:18.442226+0000","last_undegraded":"2023-07-04T18:52:18.442226+0000","last_fullsized":"2023-07-04T18:52:18.442226+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:13:52.861879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440692+0000","last_change":"2023-07-04T18:52:18.440692+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440692+0000","last_undegraded":"2023-07-04T18:52:18.440692+0000","last_fullsized":"2023-07-04T18:52:18.440692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.442502+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:52:18.442502+0000","last_peered":"2023-07-04T18:52:18.442502+0000","last_clean":"2023-07-04T18:52:18.442502+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:52:18.442502+0000","last_undegraded":"2023-07-04T18:52:18.442502+0000","last_fullsized":"2023-07-04T18:52:18.442502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:45:33.859559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.440413+0000","last_change":"2023-07-04T18:52:18.440413+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.440413+0000","last_undegraded":"2023-07-04T18:52:18.440413+0000","last_fullsized":"2023-07-04T18:52:18.440413+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.447259+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:52:18.447259+0000","last_peered":"2023-07-04T18:52:18.447259+0000","last_clean":"2023-07-04T18:52:18.447259+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:52:18.447259+0000","last_undegraded":"2023-07-04T18:52:18.447259+0000","last_fullsized":"2023-07-04T18:52:18.447259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:53:11.058829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-07-04T18:52:19.437396+0000","last_change":"2023-07-04T18:52:19.437396+0000","last_active":"2023-07-04T18:52:19.437396+0000","last_peered":"2023-07-04T18:52:19.437396+0000","last_clean":"2023-07-04T18:52:19.437396+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:19.437396+0000","last_undegraded":"2023-07-04T18:52:19.437396+0000","last_fullsized":"2023-07-04T18:52:19.437396+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-07-04T18:52:18.442368+0000","last_change":"2023-07-04T18:52:18.442368+0000","last_active":"2023-07-04T18:52:18.435765+0000","last_peered":"2023-07-04T18:52:18.435765+0000","last_clean":"2023-07-04T18:52:18.435765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-07-04T18:52:18.442368+0000","last_undegraded":"2023-07-04T18:52:18.442368+0000","last_fullsized":"2023-07-04T18:52:18.442368+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-07-04T18:52:18.439560+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:52:18.439560+0000","last_peered":"2023-07-04T18:52:18.439560+0000","last_clean":"2023-07-04T18:52:18.439560+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:52:18.439560+0000","last_undegraded":"2023-07-04T18:52:18.439560+0000","last_fullsized":"2023-07-04T18:52:18.439560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:39:26.295849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":978944,"data_stored":745347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55500000000000005}]},{"osd":1,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.68600000000000005}]},{"osd":5,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71899999999999997}]},{"osd":6,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.56000000000000005}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.621},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.504}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":372736,"data_stored":148809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.27700000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.34000000000000002}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":3.0600000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.434}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.42499999999999999}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.498},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.46500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27908,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343932,"statfs":{"total":96636764160,"available":96608186368,"internally_reserved":0,"allocated":978944,"data_stored":745347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.217},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.29199999999999998}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.311},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.32500000000000001}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.7719999999999998,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51400000000000001}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":3.7109999999999999,"5min":3.7109999999999999,"15min":3.7109999999999999},"last":0.45500000000000002}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.44700000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46800000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.217}]},{"osd":1,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28599999999999998}]},{"osd":3,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.313},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.30099999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45600000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51500000000000001}]},{"osd":6,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34899999999999998}]},{"osd":1,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.317}]},{"osd":2,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.38},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.27900000000000003}]},{"osd":4,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.503},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.46200000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.54900000000000004}]},{"osd":6,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.47099999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":385024,"data_stored":154979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.499}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.375},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.59099999999999997}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.61499999999999999}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":372736,"data_stored":148809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.50700000000000001}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.95799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:52:24.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:52:25.111 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.109+0000 7ff9eb111700 1 Processor -- start 2023-07-04T18:52:25.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 -- start start 2023-07-04T18:52:25.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 0x7ff9e40d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9e40af8f0 0x7ff9e40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e40b2790 con 0x7ff9e4146270 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e40b2a70 con 0x7ff9e40af8f0 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9eb111700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e40b2d50 con 0x7ff9e40b1860 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.110+0000 7ff9e3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9e40af8f0 0x7ff9e40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff9e40af8f0 0x7ff9e40d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46004/0 (socket says 172.21.15.77:46004) 2023-07-04T18:52:25.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e96ae700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff9e4146270 0x7ff9e41602d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53142/0 (socket says 172.21.15.77:53142) 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 -- 172.21.15.77:0/531033529 learned_addr learned my addr 172.21.15.77:0/531033529 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e8ead700 1 --2- 172.21.15.77:0/531033529 >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 0x7ff9e40d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 -- 172.21.15.77:0/531033529 >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 msgr2=0x7ff9e40d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 --2- 172.21.15.77:0/531033529 >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 0x7ff9e40d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 -- 172.21.15.77:0/531033529 >> v1:172.21.15.77:6789/0 conn(0x7ff9e4146270 legacy=0x7ff9e41602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e37fe700 1 -- 172.21.15.77:0/531033529 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005870367 0 0) 0x7ff9e40b2790 con 0x7ff9e4146270 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 -- 172.21.15.77:0/531033529 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40b3030 con 0x7ff9e40af8f0 2023-07-04T18:52:25.119 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e8ead700 1 --2- 172.21.15.77:0/531033529 >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 0x7ff9e40d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e3fff700 1 --2- 172.21.15.77:0/531033529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e40d31f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff9e408b840 tx=0x7ff9d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bacf042fef22798 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e37fe700 1 -- 172.21.15.77:0/531033529 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000b870 con 0x7ff9e40af8f0 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e37fe700 1 -- 172.21.15.77:0/531033529 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d000b9d0 con 0x7ff9e40af8f0 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.111+0000 7ff9e37fe700 1 -- 172.21.15.77:0/531033529 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000bd60 con 0x7ff9e40af8f0 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 -- 172.21.15.77:0/531033529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 msgr2=0x7ff9e40d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 --2- 172.21.15.77:0/531033529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e40d31f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff9e408b840 tx=0x7ff9d0009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 -- 172.21.15.77:0/531033529 shutdown_connections 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 --2- 172.21.15.77:0/531033529 >> v2:172.21.15.82:3300/0 conn(0x7ff9e40b1860 0x7ff9e40d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 --2- 172.21.15.77:0/531033529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e40d31f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 -- 172.21.15.77:0/531033529 >> 172.21.15.77:0/531033529 conn(0x7ff9e4071e90 msgr2=0x7ff9e409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 -- 172.21.15.77:0/531033529 shutdown_connections 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.112+0000 7ff9eb111700 1 -- 172.21.15.77:0/531033529 wait complete. 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.113+0000 7ff9eb111700 1 Processor -- start 2023-07-04T18:52:25.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.113+0000 7ff9eb111700 1 -- start start 2023-07-04T18:52:25.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff9e40b1860 0x7ff9e40cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e4146270 0x7ff9e40cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e4149940 con 0x7ff9e40af8f0 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40d01a0 con 0x7ff9e4146270 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e41469f0 con 0x7ff9e40b1860 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46006/0 (socket says 172.21.15.77:46006) 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 -- 172.21.15.77:0/4291496770 learned_addr learned my addr 172.21.15.77:0/4291496770 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e96ae700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e4146270 0x7ff9e40cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 -- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff9e40b1860 msgr2=0x7ff9e40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff9e40b1860 0x7ff9e40cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 -- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e4146270 msgr2=0x7ff9e40cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e4146270 0x7ff9e40cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 -- 172.21.15.77:0/4291496770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0009550 con 0x7ff9e40af8f0 2023-07-04T18:52:25.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e8ead700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c750 tx=0x7ff9d800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8011800 con 0x7ff9e40af8f0 2023-07-04T18:52:25.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8011960 con 0x7ff9e40af8f0 2023-07-04T18:52:25.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e4146cd0 con 0x7ff9e40af8f0 2023-07-04T18:52:25.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8011cd0 con 0x7ff9e40af8f0 2023-07-04T18:52:25.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.114+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e4147290 con 0x7ff9e40af8f0 2023-07-04T18:52:25.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.115+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e404e210 con 0x7ff9e40af8f0 2023-07-04T18:52:25.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.116+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff9d8021350 con 0x7ff9e40af8f0 2023-07-04T18:52:25.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.116+0000 7ff9e17fa700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 0x7ff9c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.116+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7ff9d8089520 con 0x7ff9e40af8f0 2023-07-04T18:52:25.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.116+0000 7ff9e3fff700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 0x7ff9c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.117+0000 7ff9e3fff700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 0x7ff9c806d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0000c00 tx=0x7ff9d0010000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.120+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff9d8054530 con 0x7ff9e40af8f0 2023-07-04T18:52:25.304 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.302+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9e40b3310 con 0x7ff9c806ace0 2023-07-04T18:52:25.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.311+0000 7ff9e17fa700 1 -- 172.21.15.77:0/4291496770 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311460 (secure 0 0 0) 0x7ff9d001e070 con 0x7ff9c806ace0 2023-07-04T18:52:25.316 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 msgr2=0x7ff9c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 0x7ff9c806d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0000c00 tx=0x7ff9d0010000 comp rx=0 tx=0).stop 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 msgr2=0x7ff9e4149400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c750 tx=0x7ff9d800cb10 comp rx=0 tx=0).stop 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 shutdown_connections 2023-07-04T18:52:25.318 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7ff9c806ace0 0x7ff9c806d190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9e40af8f0 0x7ff9e4149400 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e4146270 0x7ff9e40cfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 --2- 172.21.15.77:0/4291496770 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7ff9e40b1860 0x7ff9e40cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 >> 172.21.15.77:0/4291496770 conn(0x7ff9e4071e90 msgr2=0x7ff9e40d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 shutdown_connections 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.315+0000 7ff9eb111700 1 -- 172.21.15.77:0/4291496770 wait complete. 2023-07-04T18:52:25.319 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:52:25.328 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2023-07-04T18:52:25.192082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220136,"kb_used_data":4520,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754584,"statfs":{"total":773094113280,"available":772868694016,"internally_reserved":0,"allocated":4628480,"data_stored":2588564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001211"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470363+0000","last_change":"2023-07-04T18:52:20.487506+0000","last_active":"2023-07-04T18:52:21.470363+0000","last_peered":"2023-07-04T18:52:21.470363+0000","last_clean":"2023-07-04T18:52:21.470363+0000","last_became_active":"2023-07-04T18:52:20.487248+0000","last_became_peered":"2023-07-04T18:52:20.487248+0000","last_unstale":"2023-07-04T18:52:21.470363+0000","last_undegraded":"2023-07-04T18:52:21.470363+0000","last_fullsized":"2023-07-04T18:52:21.470363+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470386+0000","last_change":"2023-07-04T18:52:20.485291+0000","last_active":"2023-07-04T18:52:21.470386+0000","last_peered":"2023-07-04T18:52:21.470386+0000","last_clean":"2023-07-04T18:52:21.470386+0000","last_became_active":"2023-07-04T18:52:20.484107+0000","last_became_peered":"2023-07-04T18:52:20.484107+0000","last_unstale":"2023-07-04T18:52:21.470386+0000","last_undegraded":"2023-07-04T18:52:21.470386+0000","last_fullsized":"2023-07-04T18:52:21.470386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:44.397113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064060+0000","last_change":"2023-07-04T18:52:20.492813+0000","last_active":"2023-07-04T18:52:22.064060+0000","last_peered":"2023-07-04T18:52:22.064060+0000","last_clean":"2023-07-04T18:52:22.064060+0000","last_became_active":"2023-07-04T18:52:20.492538+0000","last_became_peered":"2023-07-04T18:52:20.492538+0000","last_unstale":"2023-07-04T18:52:22.064060+0000","last_undegraded":"2023-07-04T18:52:22.064060+0000","last_fullsized":"2023-07-04T18:52:22.064060+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581360+0000","last_change":"2023-07-04T18:52:20.478667+0000","last_active":"2023-07-04T18:52:21.581360+0000","last_peered":"2023-07-04T18:52:21.581360+0000","last_clean":"2023-07-04T18:52:21.581360+0000","last_became_active":"2023-07-04T18:52:20.476017+0000","last_became_peered":"2023-07-04T18:52:20.476017+0000","last_unstale":"2023-07-04T18:52:21.581360+0000","last_undegraded":"2023-07-04T18:52:21.581360+0000","last_fullsized":"2023-07-04T18:52:21.581360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581190+0000","last_change":"2023-07-04T18:52:20.475096+0000","last_active":"2023-07-04T18:52:21.581190+0000","last_peered":"2023-07-04T18:52:21.581190+0000","last_clean":"2023-07-04T18:52:21.581190+0000","last_became_active":"2023-07-04T18:52:20.474728+0000","last_became_peered":"2023-07-04T18:52:20.474728+0000","last_unstale":"2023-07-04T18:52:21.581190+0000","last_undegraded":"2023-07-04T18:52:21.581190+0000","last_fullsized":"2023-07-04T18:52:21.581190+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:06:00.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063731+0000","last_change":"2023-07-04T18:52:20.479913+0000","last_active":"2023-07-04T18:52:22.063731+0000","last_peered":"2023-07-04T18:52:22.063731+0000","last_clean":"2023-07-04T18:52:22.063731+0000","last_became_active":"2023-07-04T18:52:20.479649+0000","last_became_peered":"2023-07-04T18:52:20.479649+0000","last_unstale":"2023-07-04T18:52:22.063731+0000","last_undegraded":"2023-07-04T18:52:22.063731+0000","last_fullsized":"2023-07-04T18:52:22.063731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:10.324552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583342+0000","last_change":"2023-07-04T18:52:20.494079+0000","last_active":"2023-07-04T18:52:21.583342+0000","last_peered":"2023-07-04T18:52:21.583342+0000","last_clean":"2023-07-04T18:52:21.583342+0000","last_became_active":"2023-07-04T18:52:20.493552+0000","last_became_peered":"2023-07-04T18:52:20.493552+0000","last_unstale":"2023-07-04T18:52:21.583342+0000","last_undegraded":"2023-07-04T18:52:21.583342+0000","last_fullsized":"2023-07-04T18:52:21.583342+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:30.610249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470099+0000","last_change":"2023-07-04T18:52:20.496203+0000","last_active":"2023-07-04T18:52:21.470099+0000","last_peered":"2023-07-04T18:52:21.470099+0000","last_clean":"2023-07-04T18:52:21.470099+0000","last_became_active":"2023-07-04T18:52:20.495950+0000","last_became_peered":"2023-07-04T18:52:20.495950+0000","last_unstale":"2023-07-04T18:52:21.470099+0000","last_undegraded":"2023-07-04T18:52:21.470099+0000","last_fullsized":"2023-07-04T18:52:21.470099+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064288+0000","last_change":"2023-07-04T18:52:20.497663+0000","last_active":"2023-07-04T18:52:22.064288+0000","last_peered":"2023-07-04T18:52:22.064288+0000","last_clean":"2023-07-04T18:52:22.064288+0000","last_became_active":"2023-07-04T18:52:20.497366+0000","last_became_peered":"2023-07-04T18:52:20.497366+0000","last_unstale":"2023-07-04T18:52:22.064288+0000","last_undegraded":"2023-07-04T18:52:22.064288+0000","last_fullsized":"2023-07-04T18:52:22.064288+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583734+0000","last_change":"2023-07-04T18:52:20.494517+0000","last_active":"2023-07-04T18:52:21.583734+0000","last_peered":"2023-07-04T18:52:21.583734+0000","last_clean":"2023-07-04T18:52:21.583734+0000","last_became_active":"2023-07-04T18:52:20.493002+0000","last_became_peered":"2023-07-04T18:52:20.493002+0000","last_unstale":"2023-07-04T18:52:21.583734+0000","last_undegraded":"2023-07-04T18:52:21.583734+0000","last_fullsized":"2023-07-04T18:52:21.583734+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583902+0000","last_change":"2023-07-04T18:52:20.492866+0000","last_active":"2023-07-04T18:52:21.583902+0000","last_peered":"2023-07-04T18:52:21.583902+0000","last_clean":"2023-07-04T18:52:21.583902+0000","last_became_active":"2023-07-04T18:52:20.490309+0000","last_became_peered":"2023-07-04T18:52:20.490309+0000","last_unstale":"2023-07-04T18:52:21.583902+0000","last_undegraded":"2023-07-04T18:52:21.583902+0000","last_fullsized":"2023-07-04T18:52:21.583902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:57.519386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064222+0000","last_change":"2023-07-04T18:52:20.480329+0000","last_active":"2023-07-04T18:52:22.064222+0000","last_peered":"2023-07-04T18:52:22.064222+0000","last_clean":"2023-07-04T18:52:22.064222+0000","last_became_active":"2023-07-04T18:52:20.480036+0000","last_became_peered":"2023-07-04T18:52:20.480036+0000","last_unstale":"2023-07-04T18:52:22.064222+0000","last_undegraded":"2023-07-04T18:52:22.064222+0000","last_fullsized":"2023-07-04T18:52:22.064222+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583547+0000","last_change":"2023-07-04T18:52:20.492993+0000","last_active":"2023-07-04T18:52:21.583547+0000","last_peered":"2023-07-04T18:52:21.583547+0000","last_clean":"2023-07-04T18:52:21.583547+0000","last_became_active":"2023-07-04T18:52:20.492523+0000","last_became_peered":"2023-07-04T18:52:20.492523+0000","last_unstale":"2023-07-04T18:52:21.583547+0000","last_undegraded":"2023-07-04T18:52:21.583547+0000","last_fullsized":"2023-07-04T18:52:21.583547+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:25.636206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471448+0000","last_change":"2023-07-04T18:52:20.476182+0000","last_active":"2023-07-04T18:52:21.471448+0000","last_peered":"2023-07-04T18:52:21.471448+0000","last_clean":"2023-07-04T18:52:21.471448+0000","last_became_active":"2023-07-04T18:52:20.475870+0000","last_became_peered":"2023-07-04T18:52:20.475870+0000","last_unstale":"2023-07-04T18:52:21.471448+0000","last_undegraded":"2023-07-04T18:52:21.471448+0000","last_fullsized":"2023-07-04T18:52:21.471448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583326+0000","last_change":"2023-07-04T18:52:20.493851+0000","last_active":"2023-07-04T18:52:21.583326+0000","last_peered":"2023-07-04T18:52:21.583326+0000","last_clean":"2023-07-04T18:52:21.583326+0000","last_became_active":"2023-07-04T18:52:20.493121+0000","last_became_peered":"2023-07-04T18:52:20.493121+0000","last_unstale":"2023-07-04T18:52:21.583326+0000","last_undegraded":"2023-07-04T18:52:21.583326+0000","last_fullsized":"2023-07-04T18:52:21.583326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064111+0000","last_change":"2023-07-04T18:52:20.496287+0000","last_active":"2023-07-04T18:52:22.064111+0000","last_peered":"2023-07-04T18:52:22.064111+0000","last_clean":"2023-07-04T18:52:22.064111+0000","last_became_active":"2023-07-04T18:52:20.495989+0000","last_became_peered":"2023-07-04T18:52:20.495989+0000","last_unstale":"2023-07-04T18:52:22.064111+0000","last_undegraded":"2023-07-04T18:52:22.064111+0000","last_fullsized":"2023-07-04T18:52:22.064111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064508+0000","last_change":"2023-07-04T18:52:20.497658+0000","last_active":"2023-07-04T18:52:22.064508+0000","last_peered":"2023-07-04T18:52:22.064508+0000","last_clean":"2023-07-04T18:52:22.064508+0000","last_became_active":"2023-07-04T18:52:20.497373+0000","last_became_peered":"2023-07-04T18:52:20.497373+0000","last_unstale":"2023-07-04T18:52:22.064508+0000","last_undegraded":"2023-07-04T18:52:22.064508+0000","last_fullsized":"2023-07-04T18:52:22.064508+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:12:06.480252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471789+0000","last_change":"2023-07-04T18:52:20.486609+0000","last_active":"2023-07-04T18:52:21.471789+0000","last_peered":"2023-07-04T18:52:21.471789+0000","last_clean":"2023-07-04T18:52:21.471789+0000","last_became_active":"2023-07-04T18:52:20.486292+0000","last_became_peered":"2023-07-04T18:52:20.486292+0000","last_unstale":"2023-07-04T18:52:21.471789+0000","last_undegraded":"2023-07-04T18:52:21.471789+0000","last_fullsized":"2023-07-04T18:52:21.471789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:44:15.915109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581263+0000","last_change":"2023-07-04T18:52:20.472502+0000","last_active":"2023-07-04T18:52:21.581263+0000","last_peered":"2023-07-04T18:52:21.581263+0000","last_clean":"2023-07-04T18:52:21.581263+0000","last_became_active":"2023-07-04T18:52:20.472250+0000","last_became_peered":"2023-07-04T18:52:20.472250+0000","last_unstale":"2023-07-04T18:52:21.581263+0000","last_undegraded":"2023-07-04T18:52:21.581263+0000","last_fullsized":"2023-07-04T18:52:21.581263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:04.134366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469450+0000","last_change":"2023-07-04T18:52:20.485739+0000","last_active":"2023-07-04T18:52:21.469450+0000","last_peered":"2023-07-04T18:52:21.469450+0000","last_clean":"2023-07-04T18:52:21.469450+0000","last_became_active":"2023-07-04T18:52:20.485405+0000","last_became_peered":"2023-07-04T18:52:20.485405+0000","last_unstale":"2023-07-04T18:52:21.469450+0000","last_undegraded":"2023-07-04T18:52:21.469450+0000","last_fullsized":"2023-07-04T18:52:21.469450+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583520+0000","last_change":"2023-07-04T18:52:20.488728+0000","last_active":"2023-07-04T18:52:21.583520+0000","last_peered":"2023-07-04T18:52:21.583520+0000","last_clean":"2023-07-04T18:52:21.583520+0000","last_became_active":"2023-07-04T18:52:20.488449+0000","last_became_peered":"2023-07-04T18:52:20.488449+0000","last_unstale":"2023-07-04T18:52:21.583520+0000","last_undegraded":"2023-07-04T18:52:21.583520+0000","last_fullsized":"2023-07-04T18:52:21.583520+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:04:28.623773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471182+0000","last_change":"2023-07-04T18:52:20.475095+0000","last_active":"2023-07-04T18:52:21.471182+0000","last_peered":"2023-07-04T18:52:21.471182+0000","last_clean":"2023-07-04T18:52:21.471182+0000","last_became_active":"2023-07-04T18:52:20.474780+0000","last_became_peered":"2023-07-04T18:52:20.474780+0000","last_unstale":"2023-07-04T18:52:21.471182+0000","last_undegraded":"2023-07-04T18:52:21.471182+0000","last_fullsized":"2023-07-04T18:52:21.471182+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064410+0000","last_change":"2023-07-04T18:52:20.478932+0000","last_active":"2023-07-04T18:52:22.064410+0000","last_peered":"2023-07-04T18:52:22.064410+0000","last_clean":"2023-07-04T18:52:22.064410+0000","last_became_active":"2023-07-04T18:52:20.477175+0000","last_became_peered":"2023-07-04T18:52:20.477175+0000","last_unstale":"2023-07-04T18:52:22.064410+0000","last_undegraded":"2023-07-04T18:52:22.064410+0000","last_fullsized":"2023-07-04T18:52:22.064410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:17.945865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581066+0000","last_change":"2023-07-04T18:52:20.476929+0000","last_active":"2023-07-04T18:52:21.581066+0000","last_peered":"2023-07-04T18:52:21.581066+0000","last_clean":"2023-07-04T18:52:21.581066+0000","last_became_active":"2023-07-04T18:52:20.476625+0000","last_became_peered":"2023-07-04T18:52:20.476625+0000","last_unstale":"2023-07-04T18:52:21.581066+0000","last_undegraded":"2023-07-04T18:52:21.581066+0000","last_fullsized":"2023-07-04T18:52:21.581066+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:00:15.725242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471820+0000","last_change":"2023-07-04T18:52:20.486108+0000","last_active":"2023-07-04T18:52:21.471820+0000","last_peered":"2023-07-04T18:52:21.471820+0000","last_clean":"2023-07-04T18:52:21.471820+0000","last_became_active":"2023-07-04T18:52:20.485841+0000","last_became_peered":"2023-07-04T18:52:20.485841+0000","last_unstale":"2023-07-04T18:52:21.471820+0000","last_undegraded":"2023-07-04T18:52:21.471820+0000","last_fullsized":"2023-07-04T18:52:21.471820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:23:46.685971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064525+0000","last_change":"2023-07-04T18:52:20.492895+0000","last_active":"2023-07-04T18:52:22.064525+0000","last_peered":"2023-07-04T18:52:22.064525+0000","last_clean":"2023-07-04T18:52:22.064525+0000","last_became_active":"2023-07-04T18:52:20.492622+0000","last_became_peered":"2023-07-04T18:52:20.492622+0000","last_unstale":"2023-07-04T18:52:22.064525+0000","last_undegraded":"2023-07-04T18:52:22.064525+0000","last_fullsized":"2023-07-04T18:52:22.064525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471649+0000","last_change":"2023-07-04T18:52:20.475874+0000","last_active":"2023-07-04T18:52:21.471649+0000","last_peered":"2023-07-04T18:52:21.471649+0000","last_clean":"2023-07-04T18:52:21.471649+0000","last_became_active":"2023-07-04T18:52:20.475558+0000","last_became_peered":"2023-07-04T18:52:20.475558+0000","last_unstale":"2023-07-04T18:52:21.471649+0000","last_undegraded":"2023-07-04T18:52:21.471649+0000","last_fullsized":"2023-07-04T18:52:21.471649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:51.752371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583963+0000","last_change":"2023-07-04T18:52:20.494100+0000","last_active":"2023-07-04T18:52:21.583963+0000","last_peered":"2023-07-04T18:52:21.583963+0000","last_clean":"2023-07-04T18:52:21.583963+0000","last_became_active":"2023-07-04T18:52:20.493125+0000","last_became_peered":"2023-07-04T18:52:20.493125+0000","last_unstale":"2023-07-04T18:52:21.583963+0000","last_undegraded":"2023-07-04T18:52:21.583963+0000","last_fullsized":"2023-07-04T18:52:21.583963+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469452+0000","last_change":"2023-07-04T18:52:20.477696+0000","last_active":"2023-07-04T18:52:21.469452+0000","last_peered":"2023-07-04T18:52:21.469452+0000","last_clean":"2023-07-04T18:52:21.469452+0000","last_became_active":"2023-07-04T18:52:20.477427+0000","last_became_peered":"2023-07-04T18:52:20.477427+0000","last_unstale":"2023-07-04T18:52:21.469452+0000","last_undegraded":"2023-07-04T18:52:21.469452+0000","last_fullsized":"2023-07-04T18:52:21.469452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469469+0000","last_change":"2023-07-04T18:52:20.479863+0000","last_active":"2023-07-04T18:52:21.469469+0000","last_peered":"2023-07-04T18:52:21.469469+0000","last_clean":"2023-07-04T18:52:21.469469+0000","last_became_active":"2023-07-04T18:52:20.479580+0000","last_became_peered":"2023-07-04T18:52:20.479580+0000","last_unstale":"2023-07-04T18:52:21.469469+0000","last_undegraded":"2023-07-04T18:52:21.469469+0000","last_fullsized":"2023-07-04T18:52:21.469469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064261+0000","last_change":"2023-07-04T18:52:20.478043+0000","last_active":"2023-07-04T18:52:22.064261+0000","last_peered":"2023-07-04T18:52:22.064261+0000","last_clean":"2023-07-04T18:52:22.064261+0000","last_became_active":"2023-07-04T18:52:20.476781+0000","last_became_peered":"2023-07-04T18:52:20.476781+0000","last_unstale":"2023-07-04T18:52:22.064261+0000","last_undegraded":"2023-07-04T18:52:22.064261+0000","last_fullsized":"2023-07-04T18:52:22.064261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063720+0000","last_change":"2023-07-04T18:52:20.472516+0000","last_active":"2023-07-04T18:52:22.063720+0000","last_peered":"2023-07-04T18:52:22.063720+0000","last_clean":"2023-07-04T18:52:22.063720+0000","last_became_active":"2023-07-04T18:52:20.471963+0000","last_became_peered":"2023-07-04T18:52:20.471963+0000","last_unstale":"2023-07-04T18:52:22.063720+0000","last_undegraded":"2023-07-04T18:52:22.063720+0000","last_fullsized":"2023-07-04T18:52:22.063720+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064356+0000","last_change":"2023-07-04T18:52:20.495269+0000","last_active":"2023-07-04T18:52:22.064356+0000","last_peered":"2023-07-04T18:52:22.064356+0000","last_clean":"2023-07-04T18:52:22.064356+0000","last_became_active":"2023-07-04T18:52:20.495024+0000","last_became_peered":"2023-07-04T18:52:20.495024+0000","last_unstale":"2023-07-04T18:52:22.064356+0000","last_undegraded":"2023-07-04T18:52:22.064356+0000","last_fullsized":"2023-07-04T18:52:22.064356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:47.935529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"36'4","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471842+0000","last_change":"2023-07-04T18:52:19.456618+0000","last_active":"2023-07-04T18:52:21.471842+0000","last_peered":"2023-07-04T18:52:21.471842+0000","last_clean":"2023-07-04T18:52:21.471842+0000","last_became_active":"2023-07-04T18:52:19.456387+0000","last_became_peered":"2023-07-04T18:52:19.456387+0000","last_unstale":"2023-07-04T18:52:21.471842+0000","last_undegraded":"2023-07-04T18:52:21.471842+0000","last_fullsized":"2023-07-04T18:52:21.471842+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:55:51.483834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064499+0000","last_change":"2023-07-04T18:52:20.498353+0000","last_active":"2023-07-04T18:52:22.064499+0000","last_peered":"2023-07-04T18:52:22.064499+0000","last_clean":"2023-07-04T18:52:22.064499+0000","last_became_active":"2023-07-04T18:52:20.498137+0000","last_became_peered":"2023-07-04T18:52:20.498137+0000","last_unstale":"2023-07-04T18:52:22.064499+0000","last_undegraded":"2023-07-04T18:52:22.064499+0000","last_fullsized":"2023-07-04T18:52:22.064499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:53:25.229985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064102+0000","last_change":"2023-07-04T18:52:19.462233+0000","last_active":"2023-07-04T18:52:22.064102+0000","last_peered":"2023-07-04T18:52:22.064102+0000","last_clean":"2023-07-04T18:52:22.064102+0000","last_became_active":"2023-07-04T18:52:19.461952+0000","last_became_peered":"2023-07-04T18:52:19.461952+0000","last_unstale":"2023-07-04T18:52:22.064102+0000","last_undegraded":"2023-07-04T18:52:22.064102+0000","last_fullsized":"2023-07-04T18:52:22.064102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:43.395483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471479+0000","last_change":"2023-07-04T18:52:20.475952+0000","last_active":"2023-07-04T18:52:21.471479+0000","last_peered":"2023-07-04T18:52:21.471479+0000","last_clean":"2023-07-04T18:52:21.471479+0000","last_became_active":"2023-07-04T18:52:20.475673+0000","last_became_peered":"2023-07-04T18:52:20.475673+0000","last_unstale":"2023-07-04T18:52:21.471479+0000","last_undegraded":"2023-07-04T18:52:21.471479+0000","last_fullsized":"2023-07-04T18:52:21.471479+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064496+0000","last_change":"2023-07-04T18:52:19.458951+0000","last_active":"2023-07-04T18:52:22.064496+0000","last_peered":"2023-07-04T18:52:22.064496+0000","last_clean":"2023-07-04T18:52:22.064496+0000","last_became_active":"2023-07-04T18:52:19.458681+0000","last_became_peered":"2023-07-04T18:52:19.458681+0000","last_unstale":"2023-07-04T18:52:22.064496+0000","last_undegraded":"2023-07-04T18:52:22.064496+0000","last_fullsized":"2023-07-04T18:52:22.064496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:29.608618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470459+0000","last_change":"2023-07-04T18:52:20.486218+0000","last_active":"2023-07-04T18:52:21.470459+0000","last_peered":"2023-07-04T18:52:21.470459+0000","last_clean":"2023-07-04T18:52:21.470459+0000","last_became_active":"2023-07-04T18:52:20.485895+0000","last_became_peered":"2023-07-04T18:52:20.485895+0000","last_unstale":"2023-07-04T18:52:21.470459+0000","last_undegraded":"2023-07-04T18:52:21.470459+0000","last_fullsized":"2023-07-04T18:52:21.470459+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471467+0000","last_change":"2023-07-04T18:52:19.466273+0000","last_active":"2023-07-04T18:52:21.471467+0000","last_peered":"2023-07-04T18:52:21.471467+0000","last_clean":"2023-07-04T18:52:21.471467+0000","last_became_active":"2023-07-04T18:52:19.465971+0000","last_became_peered":"2023-07-04T18:52:19.465971+0000","last_unstale":"2023-07-04T18:52:21.471467+0000","last_undegraded":"2023-07-04T18:52:21.471467+0000","last_fullsized":"2023-07-04T18:52:21.471467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063628+0000","last_change":"2023-07-04T18:52:20.485233+0000","last_active":"2023-07-04T18:52:22.063628+0000","last_peered":"2023-07-04T18:52:22.063628+0000","last_clean":"2023-07-04T18:52:22.063628+0000","last_became_active":"2023-07-04T18:52:20.484955+0000","last_became_peered":"2023-07-04T18:52:20.484955+0000","last_unstale":"2023-07-04T18:52:22.063628+0000","last_undegraded":"2023-07-04T18:52:22.063628+0000","last_fullsized":"2023-07-04T18:52:22.063628+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:46:36.992403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583679+0000","last_change":"2023-07-04T18:52:19.459600+0000","last_active":"2023-07-04T18:52:21.583679+0000","last_peered":"2023-07-04T18:52:21.583679+0000","last_clean":"2023-07-04T18:52:21.583679+0000","last_became_active":"2023-07-04T18:52:19.459350+0000","last_became_peered":"2023-07-04T18:52:19.459350+0000","last_unstale":"2023-07-04T18:52:21.583679+0000","last_undegraded":"2023-07-04T18:52:21.583679+0000","last_fullsized":"2023-07-04T18:52:21.583679+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:01.191883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583958+0000","last_change":"2023-07-04T18:52:20.492856+0000","last_active":"2023-07-04T18:52:21.583958+0000","last_peered":"2023-07-04T18:52:21.583958+0000","last_clean":"2023-07-04T18:52:21.583958+0000","last_became_active":"2023-07-04T18:52:20.492087+0000","last_became_peered":"2023-07-04T18:52:20.492087+0000","last_unstale":"2023-07-04T18:52:21.583958+0000","last_undegraded":"2023-07-04T18:52:21.583958+0000","last_fullsized":"2023-07-04T18:52:21.583958+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064204+0000","last_change":"2023-07-04T18:52:19.464499+0000","last_active":"2023-07-04T18:52:22.064204+0000","last_peered":"2023-07-04T18:52:22.064204+0000","last_clean":"2023-07-04T18:52:22.064204+0000","last_became_active":"2023-07-04T18:52:19.464217+0000","last_became_peered":"2023-07-04T18:52:19.464217+0000","last_unstale":"2023-07-04T18:52:22.064204+0000","last_undegraded":"2023-07-04T18:52:22.064204+0000","last_fullsized":"2023-07-04T18:52:22.064204+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471012+0000","last_change":"2023-07-04T18:52:20.494161+0000","last_active":"2023-07-04T18:52:21.471012+0000","last_peered":"2023-07-04T18:52:21.471012+0000","last_clean":"2023-07-04T18:52:21.471012+0000","last_became_active":"2023-07-04T18:52:20.493910+0000","last_became_peered":"2023-07-04T18:52:20.493910+0000","last_unstale":"2023-07-04T18:52:21.471012+0000","last_undegraded":"2023-07-04T18:52:21.471012+0000","last_fullsized":"2023-07-04T18:52:21.471012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063897+0000","last_change":"2023-07-04T18:52:19.461981+0000","last_active":"2023-07-04T18:52:22.063897+0000","last_peered":"2023-07-04T18:52:22.063897+0000","last_clean":"2023-07-04T18:52:22.063897+0000","last_became_active":"2023-07-04T18:52:19.461696+0000","last_became_peered":"2023-07-04T18:52:19.461696+0000","last_unstale":"2023-07-04T18:52:22.063897+0000","last_undegraded":"2023-07-04T18:52:22.063897+0000","last_fullsized":"2023-07-04T18:52:22.063897+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:04:28.702186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470561+0000","last_change":"2023-07-04T18:52:20.481579+0000","last_active":"2023-07-04T18:52:21.470561+0000","last_peered":"2023-07-04T18:52:21.470561+0000","last_clean":"2023-07-04T18:52:21.470561+0000","last_became_active":"2023-07-04T18:52:20.480982+0000","last_became_peered":"2023-07-04T18:52:20.480982+0000","last_unstale":"2023-07-04T18:52:21.470561+0000","last_undegraded":"2023-07-04T18:52:21.470561+0000","last_fullsized":"2023-07-04T18:52:21.470561+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470811+0000","last_change":"2023-07-04T18:52:19.455607+0000","last_active":"2023-07-04T18:52:21.470811+0000","last_peered":"2023-07-04T18:52:21.470811+0000","last_clean":"2023-07-04T18:52:21.470811+0000","last_became_active":"2023-07-04T18:52:19.455315+0000","last_became_peered":"2023-07-04T18:52:19.455315+0000","last_unstale":"2023-07-04T18:52:21.470811+0000","last_undegraded":"2023-07-04T18:52:21.470811+0000","last_fullsized":"2023-07-04T18:52:21.470811+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:24.634575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581368+0000","last_change":"2023-07-04T18:52:20.476740+0000","last_active":"2023-07-04T18:52:21.581368+0000","last_peered":"2023-07-04T18:52:21.581368+0000","last_clean":"2023-07-04T18:52:21.581368+0000","last_became_active":"2023-07-04T18:52:20.475692+0000","last_became_peered":"2023-07-04T18:52:20.475692+0000","last_unstale":"2023-07-04T18:52:21.581368+0000","last_undegraded":"2023-07-04T18:52:21.581368+0000","last_fullsized":"2023-07-04T18:52:21.581368+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470439+0000","last_change":"2023-07-04T18:52:19.466605+0000","last_active":"2023-07-04T18:52:21.470439+0000","last_peered":"2023-07-04T18:52:21.470439+0000","last_clean":"2023-07-04T18:52:21.470439+0000","last_became_active":"2023-07-04T18:52:19.466312+0000","last_became_peered":"2023-07-04T18:52:19.466312+0000","last_unstale":"2023-07-04T18:52:21.470439+0000","last_undegraded":"2023-07-04T18:52:21.470439+0000","last_fullsized":"2023-07-04T18:52:21.470439+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:46:12.053717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064462+0000","last_change":"2023-07-04T18:52:20.480353+0000","last_active":"2023-07-04T18:52:22.064462+0000","last_peered":"2023-07-04T18:52:22.064462+0000","last_clean":"2023-07-04T18:52:22.064462+0000","last_became_active":"2023-07-04T18:52:20.480033+0000","last_became_peered":"2023-07-04T18:52:20.480033+0000","last_unstale":"2023-07-04T18:52:22.064462+0000","last_undegraded":"2023-07-04T18:52:22.064462+0000","last_fullsized":"2023-07-04T18:52:22.064462+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583873+0000","last_change":"2023-07-04T18:52:19.466235+0000","last_active":"2023-07-04T18:52:21.583873+0000","last_peered":"2023-07-04T18:52:21.583873+0000","last_clean":"2023-07-04T18:52:21.583873+0000","last_became_active":"2023-07-04T18:52:19.465766+0000","last_became_peered":"2023-07-04T18:52:19.465766+0000","last_unstale":"2023-07-04T18:52:21.583873+0000","last_undegraded":"2023-07-04T18:52:21.583873+0000","last_fullsized":"2023-07-04T18:52:21.583873+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063868+0000","last_change":"2023-07-04T18:52:20.485459+0000","last_active":"2023-07-04T18:52:22.063868+0000","last_peered":"2023-07-04T18:52:22.063868+0000","last_clean":"2023-07-04T18:52:22.063868+0000","last_became_active":"2023-07-04T18:52:20.485176+0000","last_became_peered":"2023-07-04T18:52:20.485176+0000","last_unstale":"2023-07-04T18:52:22.063868+0000","last_undegraded":"2023-07-04T18:52:22.063868+0000","last_fullsized":"2023-07-04T18:52:22.063868+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"36'3","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583965+0000","last_change":"2023-07-04T18:52:19.458524+0000","last_active":"2023-07-04T18:52:21.583965+0000","last_peered":"2023-07-04T18:52:21.583965+0000","last_clean":"2023-07-04T18:52:21.583965+0000","last_became_active":"2023-07-04T18:52:19.458189+0000","last_became_peered":"2023-07-04T18:52:19.458189+0000","last_unstale":"2023-07-04T18:52:21.583965+0000","last_undegraded":"2023-07-04T18:52:21.583965+0000","last_fullsized":"2023-07-04T18:52:21.583965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:16.944234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471751+0000","last_change":"2023-07-04T18:52:20.473599+0000","last_active":"2023-07-04T18:52:21.471751+0000","last_peered":"2023-07-04T18:52:21.471751+0000","last_clean":"2023-07-04T18:52:21.471751+0000","last_became_active":"2023-07-04T18:52:20.473309+0000","last_became_peered":"2023-07-04T18:52:20.473309+0000","last_unstale":"2023-07-04T18:52:21.471751+0000","last_undegraded":"2023-07-04T18:52:21.471751+0000","last_fullsized":"2023-07-04T18:52:21.471751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"36'6","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470526+0000","last_change":"2023-07-04T18:52:19.465448+0000","last_active":"2023-07-04T18:52:21.470526+0000","last_peered":"2023-07-04T18:52:21.470526+0000","last_clean":"2023-07-04T18:52:21.470526+0000","last_became_active":"2023-07-04T18:52:19.465166+0000","last_became_peered":"2023-07-04T18:52:19.465166+0000","last_unstale":"2023-07-04T18:52:21.470526+0000","last_undegraded":"2023-07-04T18:52:21.470526+0000","last_fullsized":"2023-07-04T18:52:21.470526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:39.388420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063926+0000","last_change":"2023-07-04T18:52:20.496158+0000","last_active":"2023-07-04T18:52:22.063926+0000","last_peered":"2023-07-04T18:52:22.063926+0000","last_clean":"2023-07-04T18:52:22.063926+0000","last_became_active":"2023-07-04T18:52:20.495901+0000","last_became_peered":"2023-07-04T18:52:20.495901+0000","last_unstale":"2023-07-04T18:52:22.063926+0000","last_undegraded":"2023-07-04T18:52:22.063926+0000","last_fullsized":"2023-07-04T18:52:22.063926+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:09.892279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470569+0000","last_change":"2023-07-04T18:52:19.466047+0000","last_active":"2023-07-04T18:52:21.470569+0000","last_peered":"2023-07-04T18:52:21.470569+0000","last_clean":"2023-07-04T18:52:21.470569+0000","last_became_active":"2023-07-04T18:52:19.465676+0000","last_became_peered":"2023-07-04T18:52:19.465676+0000","last_unstale":"2023-07-04T18:52:21.470569+0000","last_undegraded":"2023-07-04T18:52:21.470569+0000","last_fullsized":"2023-07-04T18:52:21.470569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:09.322921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470149+0000","last_change":"2023-07-04T18:52:20.486940+0000","last_active":"2023-07-04T18:52:21.470149+0000","last_peered":"2023-07-04T18:52:21.470149+0000","last_clean":"2023-07-04T18:52:21.470149+0000","last_became_active":"2023-07-04T18:52:20.486728+0000","last_became_peered":"2023-07-04T18:52:20.486728+0000","last_unstale":"2023-07-04T18:52:21.470149+0000","last_undegraded":"2023-07-04T18:52:21.470149+0000","last_fullsized":"2023-07-04T18:52:21.470149+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063999+0000","last_change":"2023-07-04T18:52:19.458036+0000","last_active":"2023-07-04T18:52:22.063999+0000","last_peered":"2023-07-04T18:52:22.063999+0000","last_clean":"2023-07-04T18:52:22.063999+0000","last_became_active":"2023-07-04T18:52:19.457785+0000","last_became_peered":"2023-07-04T18:52:19.457785+0000","last_unstale":"2023-07-04T18:52:22.063999+0000","last_undegraded":"2023-07-04T18:52:22.063999+0000","last_fullsized":"2023-07-04T18:52:22.063999+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:30:57.141214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064474+0000","last_change":"2023-07-04T18:52:20.479886+0000","last_active":"2023-07-04T18:52:22.064474+0000","last_peered":"2023-07-04T18:52:22.064474+0000","last_clean":"2023-07-04T18:52:22.064474+0000","last_became_active":"2023-07-04T18:52:20.479590+0000","last_became_peered":"2023-07-04T18:52:20.479590+0000","last_unstale":"2023-07-04T18:52:22.064474+0000","last_undegraded":"2023-07-04T18:52:22.064474+0000","last_fullsized":"2023-07-04T18:52:22.064474+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:15.554299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469340+0000","last_change":"2023-07-04T18:52:19.456613+0000","last_active":"2023-07-04T18:52:21.469340+0000","last_peered":"2023-07-04T18:52:21.469340+0000","last_clean":"2023-07-04T18:52:21.469340+0000","last_became_active":"2023-07-04T18:52:19.456323+0000","last_became_peered":"2023-07-04T18:52:19.456323+0000","last_unstale":"2023-07-04T18:52:21.469340+0000","last_undegraded":"2023-07-04T18:52:21.469340+0000","last_fullsized":"2023-07-04T18:52:21.469340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:03.132735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470266+0000","last_change":"2023-07-04T18:52:20.486461+0000","last_active":"2023-07-04T18:52:21.470266+0000","last_peered":"2023-07-04T18:52:21.470266+0000","last_clean":"2023-07-04T18:52:21.470266+0000","last_became_active":"2023-07-04T18:52:20.486131+0000","last_became_peered":"2023-07-04T18:52:20.486131+0000","last_unstale":"2023-07-04T18:52:21.470266+0000","last_undegraded":"2023-07-04T18:52:21.470266+0000","last_fullsized":"2023-07-04T18:52:21.470266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:01.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"36'8","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064559+0000","last_change":"2023-07-04T18:52:19.464608+0000","last_active":"2023-07-04T18:52:22.064559+0000","last_peered":"2023-07-04T18:52:22.064559+0000","last_clean":"2023-07-04T18:52:22.064559+0000","last_became_active":"2023-07-04T18:52:19.464317+0000","last_became_peered":"2023-07-04T18:52:19.464317+0000","last_unstale":"2023-07-04T18:52:22.064559+0000","last_undegraded":"2023-07-04T18:52:22.064559+0000","last_fullsized":"2023-07-04T18:52:22.064559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:52.218872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470488+0000","last_change":"2023-07-04T18:52:20.484585+0000","last_active":"2023-07-04T18:52:21.470488+0000","last_peered":"2023-07-04T18:52:21.470488+0000","last_clean":"2023-07-04T18:52:21.470488+0000","last_became_active":"2023-07-04T18:52:20.484314+0000","last_became_peered":"2023-07-04T18:52:20.484314+0000","last_unstale":"2023-07-04T18:52:21.470488+0000","last_undegraded":"2023-07-04T18:52:21.470488+0000","last_fullsized":"2023-07-04T18:52:21.470488+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:06:27.571180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583814+0000","last_change":"2023-07-04T18:52:19.458147+0000","last_active":"2023-07-04T18:52:21.583814+0000","last_peered":"2023-07-04T18:52:21.583814+0000","last_clean":"2023-07-04T18:52:21.583814+0000","last_became_active":"2023-07-04T18:52:19.457690+0000","last_became_peered":"2023-07-04T18:52:19.457690+0000","last_unstale":"2023-07-04T18:52:21.583814+0000","last_undegraded":"2023-07-04T18:52:21.583814+0000","last_fullsized":"2023-07-04T18:52:21.583814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:10.064186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583832+0000","last_change":"2023-07-04T18:52:20.492732+0000","last_active":"2023-07-04T18:52:21.583832+0000","last_peered":"2023-07-04T18:52:21.583832+0000","last_clean":"2023-07-04T18:52:21.583832+0000","last_became_active":"2023-07-04T18:52:20.490197+0000","last_became_peered":"2023-07-04T18:52:20.490197+0000","last_unstale":"2023-07-04T18:52:21.583832+0000","last_undegraded":"2023-07-04T18:52:21.583832+0000","last_fullsized":"2023-07-04T18:52:21.583832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471601+0000","last_change":"2023-07-04T18:52:19.455963+0000","last_active":"2023-07-04T18:52:21.471601+0000","last_peered":"2023-07-04T18:52:21.471601+0000","last_clean":"2023-07-04T18:52:21.471601+0000","last_became_active":"2023-07-04T18:52:19.455693+0000","last_became_peered":"2023-07-04T18:52:19.455693+0000","last_unstale":"2023-07-04T18:52:21.471601+0000","last_undegraded":"2023-07-04T18:52:21.471601+0000","last_fullsized":"2023-07-04T18:52:21.471601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471710+0000","last_change":"2023-07-04T18:52:20.479233+0000","last_active":"2023-07-04T18:52:21.471710+0000","last_peered":"2023-07-04T18:52:21.471710+0000","last_clean":"2023-07-04T18:52:21.471710+0000","last_became_active":"2023-07-04T18:52:20.478983+0000","last_became_peered":"2023-07-04T18:52:20.478983+0000","last_unstale":"2023-07-04T18:52:21.471710+0000","last_undegraded":"2023-07-04T18:52:21.471710+0000","last_fullsized":"2023-07-04T18:52:21.471710+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:28:52.986168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581163+0000","last_change":"2023-07-04T18:52:19.462190+0000","last_active":"2023-07-04T18:52:21.581163+0000","last_peered":"2023-07-04T18:52:21.581163+0000","last_clean":"2023-07-04T18:52:21.581163+0000","last_became_active":"2023-07-04T18:52:19.461906+0000","last_became_peered":"2023-07-04T18:52:19.461906+0000","last_unstale":"2023-07-04T18:52:21.581163+0000","last_undegraded":"2023-07-04T18:52:21.581163+0000","last_fullsized":"2023-07-04T18:52:21.581163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:48.049300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581307+0000","last_change":"2023-07-04T18:52:20.479758+0000","last_active":"2023-07-04T18:52:21.581307+0000","last_peered":"2023-07-04T18:52:21.581307+0000","last_clean":"2023-07-04T18:52:21.581307+0000","last_became_active":"2023-07-04T18:52:20.479319+0000","last_became_peered":"2023-07-04T18:52:20.479319+0000","last_unstale":"2023-07-04T18:52:21.581307+0000","last_undegraded":"2023-07-04T18:52:21.581307+0000","last_fullsized":"2023-07-04T18:52:21.581307+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063995+0000","last_change":"2023-07-04T18:52:19.457634+0000","last_active":"2023-07-04T18:52:22.063995+0000","last_peered":"2023-07-04T18:52:22.063995+0000","last_clean":"2023-07-04T18:52:22.063995+0000","last_became_active":"2023-07-04T18:52:19.457341+0000","last_became_peered":"2023-07-04T18:52:19.457341+0000","last_unstale":"2023-07-04T18:52:22.063995+0000","last_undegraded":"2023-07-04T18:52:22.063995+0000","last_fullsized":"2023-07-04T18:52:22.063995+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:35:03.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064181+0000","last_change":"2023-07-04T18:52:20.483273+0000","last_active":"2023-07-04T18:52:22.064181+0000","last_peered":"2023-07-04T18:52:22.064181+0000","last_clean":"2023-07-04T18:52:22.064181+0000","last_became_active":"2023-07-04T18:52:20.480069+0000","last_became_peered":"2023-07-04T18:52:20.480069+0000","last_unstale":"2023-07-04T18:52:22.064181+0000","last_undegraded":"2023-07-04T18:52:22.064181+0000","last_fullsized":"2023-07-04T18:52:22.064181+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581171+0000","last_change":"2023-07-04T18:52:19.458392+0000","last_active":"2023-07-04T18:52:21.581171+0000","last_peered":"2023-07-04T18:52:21.581171+0000","last_clean":"2023-07-04T18:52:21.581171+0000","last_became_active":"2023-07-04T18:52:19.458046+0000","last_became_peered":"2023-07-04T18:52:19.458046+0000","last_unstale":"2023-07-04T18:52:21.581171+0000","last_undegraded":"2023-07-04T18:52:21.581171+0000","last_fullsized":"2023-07-04T18:52:21.581171+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:40:12.555311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063694+0000","last_change":"2023-07-04T18:52:20.482831+0000","last_active":"2023-07-04T18:52:22.063694+0000","last_peered":"2023-07-04T18:52:22.063694+0000","last_clean":"2023-07-04T18:52:22.063694+0000","last_became_active":"2023-07-04T18:52:20.482494+0000","last_became_peered":"2023-07-04T18:52:20.482494+0000","last_unstale":"2023-07-04T18:52:22.063694+0000","last_undegraded":"2023-07-04T18:52:22.063694+0000","last_fullsized":"2023-07-04T18:52:22.063694+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:45:42.643724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581048+0000","last_change":"2023-07-04T18:52:19.456664+0000","last_active":"2023-07-04T18:52:21.581048+0000","last_peered":"2023-07-04T18:52:21.581048+0000","last_clean":"2023-07-04T18:52:21.581048+0000","last_became_active":"2023-07-04T18:52:19.456374+0000","last_became_peered":"2023-07-04T18:52:19.456374+0000","last_unstale":"2023-07-04T18:52:21.581048+0000","last_undegraded":"2023-07-04T18:52:21.581048+0000","last_fullsized":"2023-07-04T18:52:21.581048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:04:05.770013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583383+0000","last_change":"2023-07-04T18:52:20.496556+0000","last_active":"2023-07-04T18:52:21.583383+0000","last_peered":"2023-07-04T18:52:21.583383+0000","last_clean":"2023-07-04T18:52:21.583383+0000","last_became_active":"2023-07-04T18:52:20.495594+0000","last_became_peered":"2023-07-04T18:52:20.495594+0000","last_unstale":"2023-07-04T18:52:21.583383+0000","last_undegraded":"2023-07-04T18:52:21.583383+0000","last_fullsized":"2023-07-04T18:52:21.583383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:29:33.232347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064681+0000","last_change":"2023-07-04T18:52:19.466675+0000","last_active":"2023-07-04T18:52:22.064681+0000","last_peered":"2023-07-04T18:52:22.064681+0000","last_clean":"2023-07-04T18:52:22.064681+0000","last_became_active":"2023-07-04T18:52:19.466465+0000","last_became_peered":"2023-07-04T18:52:19.466465+0000","last_unstale":"2023-07-04T18:52:22.064681+0000","last_undegraded":"2023-07-04T18:52:22.064681+0000","last_fullsized":"2023-07-04T18:52:22.064681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:38:13.573589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470404+0000","last_change":"2023-07-04T18:52:20.481606+0000","last_active":"2023-07-04T18:52:21.470404+0000","last_peered":"2023-07-04T18:52:21.470404+0000","last_clean":"2023-07-04T18:52:21.470404+0000","last_became_active":"2023-07-04T18:52:20.481341+0000","last_became_peered":"2023-07-04T18:52:20.481341+0000","last_unstale":"2023-07-04T18:52:21.470404+0000","last_undegraded":"2023-07-04T18:52:21.470404+0000","last_fullsized":"2023-07-04T18:52:21.470404+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:58.532607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581050+0000","last_change":"2023-07-04T18:52:19.457999+0000","last_active":"2023-07-04T18:52:21.581050+0000","last_peered":"2023-07-04T18:52:21.581050+0000","last_clean":"2023-07-04T18:52:21.581050+0000","last_became_active":"2023-07-04T18:52:19.457744+0000","last_became_peered":"2023-07-04T18:52:19.457744+0000","last_unstale":"2023-07-04T18:52:21.581050+0000","last_undegraded":"2023-07-04T18:52:21.581050+0000","last_fullsized":"2023-07-04T18:52:21.581050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:08:10.842309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063778+0000","last_change":"2023-07-04T18:52:20.485157+0000","last_active":"2023-07-04T18:52:22.063778+0000","last_peered":"2023-07-04T18:52:22.063778+0000","last_clean":"2023-07-04T18:52:22.063778+0000","last_became_active":"2023-07-04T18:52:20.484906+0000","last_became_peered":"2023-07-04T18:52:20.484906+0000","last_unstale":"2023-07-04T18:52:22.063778+0000","last_undegraded":"2023-07-04T18:52:22.063778+0000","last_fullsized":"2023-07-04T18:52:22.063778+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064086+0000","last_change":"2023-07-04T18:52:19.461058+0000","last_active":"2023-07-04T18:52:22.064086+0000","last_peered":"2023-07-04T18:52:22.064086+0000","last_clean":"2023-07-04T18:52:22.064086+0000","last_became_active":"2023-07-04T18:52:19.460786+0000","last_became_peered":"2023-07-04T18:52:19.460786+0000","last_unstale":"2023-07-04T18:52:22.064086+0000","last_undegraded":"2023-07-04T18:52:22.064086+0000","last_fullsized":"2023-07-04T18:52:22.064086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:57.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064059+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:52:22.064059+0000","last_peered":"2023-07-04T18:52:22.064059+0000","last_clean":"2023-07-04T18:52:22.064059+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:52:22.064059+0000","last_undegraded":"2023-07-04T18:52:22.064059+0000","last_fullsized":"2023-07-04T18:52:22.064059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:45:06.368879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470452+0000","last_change":"2023-07-04T18:52:20.491037+0000","last_active":"2023-07-04T18:52:21.470452+0000","last_peered":"2023-07-04T18:52:21.470452+0000","last_clean":"2023-07-04T18:52:21.470452+0000","last_became_active":"2023-07-04T18:52:20.490723+0000","last_became_peered":"2023-07-04T18:52:20.490723+0000","last_unstale":"2023-07-04T18:52:21.470452+0000","last_undegraded":"2023-07-04T18:52:21.470452+0000","last_fullsized":"2023-07-04T18:52:21.470452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063944+0000","last_change":"2023-07-04T18:52:19.462323+0000","last_active":"2023-07-04T18:52:22.063944+0000","last_peered":"2023-07-04T18:52:22.063944+0000","last_clean":"2023-07-04T18:52:22.063944+0000","last_became_active":"2023-07-04T18:52:19.462057+0000","last_became_peered":"2023-07-04T18:52:19.462057+0000","last_unstale":"2023-07-04T18:52:22.063944+0000","last_undegraded":"2023-07-04T18:52:22.063944+0000","last_fullsized":"2023-07-04T18:52:22.063944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:00.356116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583488+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:52:21.583488+0000","last_peered":"2023-07-04T18:52:21.583488+0000","last_clean":"2023-07-04T18:52:21.583488+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:52:21.583488+0000","last_undegraded":"2023-07-04T18:52:21.583488+0000","last_fullsized":"2023-07-04T18:52:21.583488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:21:22.798814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064397+0000","last_change":"2023-07-04T18:52:20.484156+0000","last_active":"2023-07-04T18:52:22.064397+0000","last_peered":"2023-07-04T18:52:22.064397+0000","last_clean":"2023-07-04T18:52:22.064397+0000","last_became_active":"2023-07-04T18:52:20.483878+0000","last_became_peered":"2023-07-04T18:52:20.483878+0000","last_unstale":"2023-07-04T18:52:22.064397+0000","last_undegraded":"2023-07-04T18:52:22.064397+0000","last_fullsized":"2023-07-04T18:52:22.064397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581165+0000","last_change":"2023-07-04T18:52:19.460102+0000","last_active":"2023-07-04T18:52:21.581165+0000","last_peered":"2023-07-04T18:52:21.581165+0000","last_clean":"2023-07-04T18:52:21.581165+0000","last_became_active":"2023-07-04T18:52:19.459830+0000","last_became_peered":"2023-07-04T18:52:19.459830+0000","last_unstale":"2023-07-04T18:52:21.581165+0000","last_undegraded":"2023-07-04T18:52:21.581165+0000","last_fullsized":"2023-07-04T18:52:21.581165+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:17:54.321667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581016+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:52:21.581016+0000","last_peered":"2023-07-04T18:52:21.581016+0000","last_clean":"2023-07-04T18:52:21.581016+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:52:21.581016+0000","last_undegraded":"2023-07-04T18:52:21.581016+0000","last_fullsized":"2023-07-04T18:52:21.581016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:13:46.966327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583836+0000","last_change":"2023-07-04T18:52:20.489978+0000","last_active":"2023-07-04T18:52:21.583836+0000","last_peered":"2023-07-04T18:52:21.583836+0000","last_clean":"2023-07-04T18:52:21.583836+0000","last_became_active":"2023-07-04T18:52:20.489690+0000","last_became_peered":"2023-07-04T18:52:20.489690+0000","last_unstale":"2023-07-04T18:52:21.583836+0000","last_undegraded":"2023-07-04T18:52:21.583836+0000","last_fullsized":"2023-07-04T18:52:21.583836+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064479+0000","last_change":"2023-07-04T18:52:19.469850+0000","last_active":"2023-07-04T18:52:22.064479+0000","last_peered":"2023-07-04T18:52:22.064479+0000","last_clean":"2023-07-04T18:52:22.064479+0000","last_became_active":"2023-07-04T18:52:19.469546+0000","last_became_peered":"2023-07-04T18:52:19.469546+0000","last_unstale":"2023-07-04T18:52:22.064479+0000","last_undegraded":"2023-07-04T18:52:22.064479+0000","last_fullsized":"2023-07-04T18:52:22.064479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:28:10.562186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583631+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:52:21.583631+0000","last_peered":"2023-07-04T18:52:21.583631+0000","last_clean":"2023-07-04T18:52:21.583631+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:52:21.583631+0000","last_undegraded":"2023-07-04T18:52:21.583631+0000","last_fullsized":"2023-07-04T18:52:21.583631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:49.853323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":197,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064304+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:52:22.064304+0000","last_peered":"2023-07-04T18:52:22.064304+0000","last_clean":"2023-07-04T18:52:22.064304+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:52:22.064304+0000","last_undegraded":"2023-07-04T18:52:22.064304+0000","last_fullsized":"2023-07-04T18:52:22.064304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:43.685172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063939+0000","last_change":"2023-07-04T18:52:20.497660+0000","last_active":"2023-07-04T18:52:22.063939+0000","last_peered":"2023-07-04T18:52:22.063939+0000","last_clean":"2023-07-04T18:52:22.063939+0000","last_became_active":"2023-07-04T18:52:20.497337+0000","last_became_peered":"2023-07-04T18:52:20.497337+0000","last_unstale":"2023-07-04T18:52:22.063939+0000","last_undegraded":"2023-07-04T18:52:22.063939+0000","last_fullsized":"2023-07-04T18:52:22.063939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:20:18.389953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583996+0000","last_change":"2023-07-04T18:52:19.461997+0000","last_active":"2023-07-04T18:52:21.583996+0000","last_peered":"2023-07-04T18:52:21.583996+0000","last_clean":"2023-07-04T18:52:21.583996+0000","last_became_active":"2023-07-04T18:52:19.461815+0000","last_became_peered":"2023-07-04T18:52:19.461815+0000","last_unstale":"2023-07-04T18:52:21.583996+0000","last_undegraded":"2023-07-04T18:52:21.583996+0000","last_fullsized":"2023-07-04T18:52:21.583996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:14.552668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064565+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:52:22.064565+0000","last_peered":"2023-07-04T18:52:22.064565+0000","last_clean":"2023-07-04T18:52:22.064565+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:52:22.064565+0000","last_undegraded":"2023-07-04T18:52:22.064565+0000","last_fullsized":"2023-07-04T18:52:22.064565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:01:20.555239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063962+0000","last_change":"2023-07-04T18:52:20.477535+0000","last_active":"2023-07-04T18:52:22.063962+0000","last_peered":"2023-07-04T18:52:22.063962+0000","last_clean":"2023-07-04T18:52:22.063962+0000","last_became_active":"2023-07-04T18:52:20.477247+0000","last_became_peered":"2023-07-04T18:52:20.477247+0000","last_unstale":"2023-07-04T18:52:22.063962+0000","last_undegraded":"2023-07-04T18:52:22.063962+0000","last_fullsized":"2023-07-04T18:52:22.063962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:00:07.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469531+0000","last_change":"2023-07-04T18:52:19.455490+0000","last_active":"2023-07-04T18:52:21.469531+0000","last_peered":"2023-07-04T18:52:21.469531+0000","last_clean":"2023-07-04T18:52:21.469531+0000","last_became_active":"2023-07-04T18:52:19.455224+0000","last_became_peered":"2023-07-04T18:52:19.455224+0000","last_unstale":"2023-07-04T18:52:21.469531+0000","last_undegraded":"2023-07-04T18:52:21.469531+0000","last_fullsized":"2023-07-04T18:52:21.469531+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:53:35.991529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064762+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:52:22.064762+0000","last_peered":"2023-07-04T18:52:22.064762+0000","last_clean":"2023-07-04T18:52:22.064762+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:52:22.064762+0000","last_undegraded":"2023-07-04T18:52:22.064762+0000","last_fullsized":"2023-07-04T18:52:22.064762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:26:23.252619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471172+0000","last_change":"2023-07-04T18:52:20.487224+0000","last_active":"2023-07-04T18:52:21.471172+0000","last_peered":"2023-07-04T18:52:21.471172+0000","last_clean":"2023-07-04T18:52:21.471172+0000","last_became_active":"2023-07-04T18:52:20.486638+0000","last_became_peered":"2023-07-04T18:52:20.486638+0000","last_unstale":"2023-07-04T18:52:21.471172+0000","last_undegraded":"2023-07-04T18:52:21.471172+0000","last_fullsized":"2023-07-04T18:52:21.471172+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:02.193514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470860+0000","last_change":"2023-07-04T18:52:19.465554+0000","last_active":"2023-07-04T18:52:21.470860+0000","last_peered":"2023-07-04T18:52:21.470860+0000","last_clean":"2023-07-04T18:52:21.470860+0000","last_became_active":"2023-07-04T18:52:19.465268+0000","last_became_peered":"2023-07-04T18:52:19.465268+0000","last_unstale":"2023-07-04T18:52:21.470860+0000","last_undegraded":"2023-07-04T18:52:21.470860+0000","last_fullsized":"2023-07-04T18:52:21.470860+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:22:24.930028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064543+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:52:22.064543+0000","last_peered":"2023-07-04T18:52:22.064543+0000","last_clean":"2023-07-04T18:52:22.064543+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:52:22.064543+0000","last_undegraded":"2023-07-04T18:52:22.064543+0000","last_fullsized":"2023-07-04T18:52:22.064543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:28:30.099502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064698+0000","last_change":"2023-07-04T18:52:20.479835+0000","last_active":"2023-07-04T18:52:22.064698+0000","last_peered":"2023-07-04T18:52:22.064698+0000","last_clean":"2023-07-04T18:52:22.064698+0000","last_became_active":"2023-07-04T18:52:20.479561+0000","last_became_peered":"2023-07-04T18:52:20.479561+0000","last_unstale":"2023-07-04T18:52:22.064698+0000","last_undegraded":"2023-07-04T18:52:22.064698+0000","last_fullsized":"2023-07-04T18:52:22.064698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:59:27.432907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470121+0000","last_change":"2023-07-04T18:52:19.461537+0000","last_active":"2023-07-04T18:52:21.470121+0000","last_peered":"2023-07-04T18:52:21.470121+0000","last_clean":"2023-07-04T18:52:21.470121+0000","last_became_active":"2023-07-04T18:52:19.461279+0000","last_became_peered":"2023-07-04T18:52:19.461279+0000","last_unstale":"2023-07-04T18:52:21.470121+0000","last_undegraded":"2023-07-04T18:52:21.470121+0000","last_fullsized":"2023-07-04T18:52:21.470121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:08.890649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471908+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:52:21.471908+0000","last_peered":"2023-07-04T18:52:21.471908+0000","last_clean":"2023-07-04T18:52:21.471908+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:52:21.471908+0000","last_undegraded":"2023-07-04T18:52:21.471908+0000","last_fullsized":"2023-07-04T18:52:21.471908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:11:33.840191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":1032192,"data_stored":767390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.66000000000000003}]},{"osd":1,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":0.68300000000000005}]},{"osd":2,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.55100000000000005}]},{"osd":3,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.61799999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.497},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59699999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63}]},{"osd":6,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53200000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54700000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.64600000000000002}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.57999999999999996}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.434},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.51200000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.27200000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.371}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":3.0600000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.626}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.68300000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":1019904,"data_stored":765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.441}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.7719999999999998,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.751},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69699999999999995}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":3.7109999999999999,"5min":3.7109999999999999,"15min":3.7109999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.79600000000000004}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.72099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.371},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.313}]},{"osd":1,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.35899999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.34499999999999997}]},{"osd":4,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.436}]},{"osd":5,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.49099999999999999}]},{"osd":6,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48099999999999998}]},{"osd":7,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":438272,"data_stored":176986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.315}]},{"osd":1,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27600000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.54200000000000004}]},{"osd":5,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.65600000000000003}]},{"osd":6,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.748},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70599999999999996}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.65100000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.84999999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.82699999999999996}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.77700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.50700000000000001}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.95799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:52:25.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-04T18:52:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.487+0000 7fad2d23d700 1 Processor -- start 2023-07-04T18:52:25.489 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 -- start start 2023-07-04T18:52:25.490 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fad280719f0 0x7fad2806fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad2815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad281610c0 con 0x7fad2806d140 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2815dce0 con 0x7fad280719f0 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2d23d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad28157190 con 0x7fad2806e390 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2759e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fad2806d140 0x7fad28069ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53156/0 (socket says 172.21.15.77:53156) 2023-07-04T18:52:25.491 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.488+0000 7fad2759e700 1 -- 172.21.15.77:0/2204529671 learned_addr learned my addr 172.21.15.77:0/2204529671 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 --2- 172.21.15.77:0/2204529671 >> v2:172.21.15.82:3300/0 conn(0x7fad280719f0 0x7fad2806fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad2659c700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad2815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814363979 0 0) 0x7fad281610c0 con 0x7fad2806d140 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad100035b0 con 0x7fad2806d140 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 -- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 msgr2=0x7fad2815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad2815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 -- 172.21.15.77:0/2204529671 >> v1:172.21.15.77:6789/0 conn(0x7fad2806d140 legacy=0x7fad28069ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.492 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528343820 0 0) 0x7fad100035b0 con 0x7fad2806d140 2023-07-04T18:52:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 -- 172.21.15.77:0/2204529671 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad281604b0 con 0x7fad280719f0 2023-07-04T18:52:25.493 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.489+0000 7fad26d9d700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2806fda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18009840 tx=0x7fad18009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2e33558f9fdc201 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200019f0 con 0x7fad280719f0 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad18004030 con 0x7fad280719f0 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad25d9b700 1 -- 172.21.15.77:0/2204529671 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad180043c0 con 0x7fad280719f0 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 -- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 msgr2=0x7fad2806fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2806fda0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fad18009840 tx=0x7fad18009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:25.494 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 -- 172.21.15.77:0/2204529671 shutdown_connections 2023-07-04T18:52:25.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2806fda0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 --2- 172.21.15.77:0/2204529671 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad2815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.490+0000 7fad2d23d700 1 -- 172.21.15.77:0/2204529671 >> 172.21.15.77:0/2204529671 conn(0x7fad280cc8b0 msgr2=0x7fad280cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:25.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.491+0000 7fad2d23d700 1 -- 172.21.15.77:0/2204529671 shutdown_connections 2023-07-04T18:52:25.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.491+0000 7fad2d23d700 1 -- 172.21.15.77:0/2204529671 wait complete. 2023-07-04T18:52:25.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.492+0000 7fad2d23d700 1 Processor -- start 2023-07-04T18:52:25.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.492+0000 7fad2d23d700 1 -- start start 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad281539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2815e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad28153ee0 con 0x7fad2806d140 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad281543d0 con 0x7fad280719f0 2023-07-04T18:52:25.498 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2d23d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2815de20 con 0x7fad2806e390 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46020/0 (socket says 172.21.15.77:46020) 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 -- 172.21.15.77:0/1741252037 learned_addr learned my addr 172.21.15.77:0/1741252037 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2759e700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad2659c700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad281539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 -- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 msgr2=0x7fad281539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.499 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad281539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 -- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 msgr2=0x7fad2815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 -- 172.21.15.77:0/1741252037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad180094a0 con 0x7fad2806d140 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.493+0000 7fad26d9d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fad180090b0 tx=0x7fad18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.494+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18003ae0 con 0x7fad2806d140 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.494+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1801f070 con 0x7fad2806d140 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.494+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2815fad0 con 0x7fad2806d140 2023-07-04T18:52:25.500 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.494+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18003ea0 con 0x7fad2806d140 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.494+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2815e070 con 0x7fad2806d140 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.495+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2804e210 con 0x7fad2806d140 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.495+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fad18021030 con 0x7fad2806d140 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.495+0000 7fad0f7fe700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 0x7fad1406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.496+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fad1808a5b0 con 0x7fad2806d140 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.496+0000 7fad2659c700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 0x7fad1406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.501 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.496+0000 7fad2659c700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 0x7fad1406d190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad28154af0 tx=0x7fad1c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.502 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.500+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fad180556f0 con 0x7fad2806d140 2023-07-04T18:52:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.764+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fad2806b450 con 0x7fad2806d140 2023-07-04T18:52:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.764+0000 7fad0f7fe700 1 -- 172.21.15.77:0/1741252037 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7fad18058d10 con 0x7fad2806d140 2023-07-04T18:52:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:25.766 INFO:teuthology.orchestra.run.smithi077.stdout:{"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":88,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":37,"num_osds":8,"num_up_osds":8,"osd_up_since":1688496669,"num_in_osds":8,"osd_in_since":1688496660,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":105}],"num_pgs":105,"num_pools":4,"num_objects":24,"data_bytes":592730,"bytes_used":225419264,"bytes_avail":772868694016,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5136}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-07-04T18:52:09.188664+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-07-04T18:52:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 msgr2=0x7fad1406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 0x7fad1406d190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad28154af0 tx=0x7fad1c009210 comp rx=0 tx=0).stop 2023-07-04T18:52:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 msgr2=0x7fad281e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fad180090b0 tx=0x7fad18004750 comp rx=0 tx=0).stop 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 shutdown_connections 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fad1406ace0 0x7fad1406d190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad2806d140 0x7fad281e6130 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad280719f0 0x7fad2815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 --2- 172.21.15.77:0/1741252037 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fad2806e390 0x7fad281539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.766+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 >> 172.21.15.77:0/1741252037 conn(0x7fad280cc8b0 msgr2=0x7fad2806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.767+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 shutdown_connections 2023-07-04T18:52:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.767+0000 7fad2d23d700 1 -- 172.21.15.77:0/1741252037 wait complete. 2023-07-04T18:52:25.781 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-07-04T18:52:25.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:52:25.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.900+0000 7f9d1d47c700 1 Processor -- start 2023-07-04T18:52:25.902 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.900+0000 7f9d1d47c700 1 -- start start 2023-07-04T18:52:25.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d1d47c700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f9d1815c640 0x7f9d180cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d1d47c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d180cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d1d47c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d180717f0 con 0x7f9d1815ca10 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d1d47c700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d180cd290 con 0x7f9d1815c640 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d1d47c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180ccfb0 con 0x7f9d18070f80 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d167fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9d1815ca10 0x7f9d18070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53168/0 (socket says 172.21.15.77:53168) 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d167fc700 1 -- 172.21.15.77:0/2897496023 learned_addr learned my addr 172.21.15.77:0/2897496023 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:25.906 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d177fe700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d180cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861486623 0 0) 0x7f9d180717f0 con 0x7f9d1815ca10 2023-07-04T18:52:25.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.901+0000 7f9d16ffd700 1 --2- 172.21.15.77:0/2897496023 >> v2:172.21.15.82:3300/0 conn(0x7f9d1815c640 0x7f9d180cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.907 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d000035b0 con 0x7f9d1815ca10 2023-07-04T18:52:25.909 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1004865712 0 0) 0x7f9d000035b0 con 0x7f9d1815ca10 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 msgr2=0x7f9d180cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d180cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 >> v2:172.21.15.82:3300/0 conn(0x7f9d1815c640 msgr2=0x7f9d180cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 --2- 172.21.15.77:0/2897496023 >> v2:172.21.15.82:3300/0 conn(0x7f9d1815c640 0x7f9d180cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d180ce640 con 0x7f9d1815ca10 2023-07-04T18:52:25.910 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f9d08001670 con 0x7f9d1815ca10 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 >> v1:172.21.15.77:6789/0 conn(0x7f9d1815ca10 legacy=0x7f9d18070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d00003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d00004d90 0x7f9d00005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180cd290 con 0x7f9d00003f80 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180ccfb0 con 0x7f9d00004d90 2023-07-04T18:52:25.911 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0000baf0 con 0x7f9d18070f80 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d16ffd700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d00003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d177fe700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d00004d90 0x7f9d00005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 -- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 msgr2=0x7f9d00003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d18070f80 0x7f9d00003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 -- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d00004d90 msgr2=0x7f9d00005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:25.912 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d00004d90 0x7f9d00005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 -- 172.21.15.77:0/2897496023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d000035b0 con 0x7f9d00003f80 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.902+0000 7f9d167fc700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d00004850 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9d08001050 tx=0x7f9d00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ff6d4be8c3d015 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d08007c60 con 0x7f9d00003f80 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0800cdc0 con 0x7f9d00003f80 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d15ffb700 1 -- 172.21.15.77:0/2897496023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d08005ba0 con 0x7f9d00003f80 2023-07-04T18:52:25.913 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 -- 172.21.15.77:0/2897496023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 msgr2=0x7f9d00004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/2897496023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d00004850 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9d08001050 tx=0x7f9d00005880 comp rx=0 tx=0).stop 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d177fe700 1 -- 172.21.15.77:0/2897496023 reap_dead start 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 -- 172.21.15.77:0/2897496023 shutdown_connections 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 -- 172.21.15.77:0/2897496023 >> 172.21.15.77:0/2897496023 conn(0x7f9d1806bba0 msgr2=0x7f9d180b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 -- 172.21.15.77:0/2897496023 shutdown_connections 2023-07-04T18:52:25.914 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.903+0000 7f9d1d47c700 1 -- 172.21.15.77:0/2897496023 wait complete. 2023-07-04T18:52:25.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.904+0000 7f9d1d47c700 1 Processor -- start 2023-07-04T18:52:25.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 -- start start 2023-07-04T18:52:25.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d1814cf30 0x7f9d180c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.917 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d180ac8b0 0x7f9d180acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180ad290 con 0x7f9d00003f80 2023-07-04T18:52:25.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180c99e0 con 0x7f9d1814cf30 2023-07-04T18:52:25.918 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d1d47c700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1814d3a0 con 0x7f9d180ac8b0 2023-07-04T18:52:25.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.919 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46040/0 (socket says 172.21.15.77:46040) 2023-07-04T18:52:25.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.905+0000 7f9d16ffd700 1 -- 172.21.15.77:0/4188620771 learned_addr learned my addr 172.21.15.77:0/4188620771 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:25.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d177fe700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d180ac8b0 0x7f9d180acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.920 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d167fc700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d1814cf30 0x7f9d180c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 -- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d180ac8b0 msgr2=0x7f9d180acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d180ac8b0 0x7f9d180acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 -- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d1814cf30 msgr2=0x7f9d180c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d1814cf30 0x7f9d180c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 -- 172.21.15.77:0/4188620771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d180c81a0 con 0x7f9d00003f80 2023-07-04T18:52:25.921 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d16ffd700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c011aa0 tx=0x7f9d0c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00f980 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0c00fb00 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00feb0 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d180c8480 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.906+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d180c8a40 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.907+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d180cf5e0 con 0x7f9d00003f80 2023-07-04T18:52:25.922 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.907+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9d0c014440 con 0x7f9d00003f80 2023-07-04T18:52:25.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.908+0000 7f9cff7fe700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 0x7f9d0406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:25.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.908+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f9d0c089be0 con 0x7f9d00003f80 2023-07-04T18:52:25.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.908+0000 7f9d167fc700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 0x7f9d0406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:25.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.908+0000 7f9d167fc700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 0x7f9d0406d190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9d08015040 tx=0x7f9d0800a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:25.923 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:25.912+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9d0c058920 con 0x7f9d00003f80 2023-07-04T18:52:26.097 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.095+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d180ce920 con 0x7f9d0406ace0 2023-07-04T18:52:26.106 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.104+0000 7f9cff7fe700 1 -- 172.21.15.77:0/4188620771 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311460 (secure 0 0 0) 0x7f9d08007b80 con 0x7f9d0406ace0 2023-07-04T18:52:26.110 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:26.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 msgr2=0x7f9d0406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:26.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 0x7f9d0406d190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9d08015040 tx=0x7f9d0800a000 comp rx=0 tx=0).stop 2023-07-04T18:52:26.112 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 msgr2=0x7f9d180c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:26.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c011aa0 tx=0x7f9d0c011e60 comp rx=0 tx=0).stop 2023-07-04T18:52:26.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 shutdown_connections 2023-07-04T18:52:26.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f9d0406ace0 0x7f9d0406d190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d00003f80 0x7f9d180c7ae0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f9d180ac8b0 0x7f9d180acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 --2- 172.21.15.77:0/4188620771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d1814cf30 0x7f9d180c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 >> 172.21.15.77:0/4188620771 conn(0x7f9d1806bba0 msgr2=0x7f9d1815bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 shutdown_connections 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:26.107+0000 7f9d1d47c700 1 -- 172.21.15.77:0/4188620771 wait complete. 2023-07-04T18:52:26.115 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:52:26.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2023-07-04T18:52:25.192082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220136,"kb_used_data":4520,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754584,"statfs":{"total":773094113280,"available":772868694016,"internally_reserved":0,"allocated":4628480,"data_stored":2588564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001211"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470363+0000","last_change":"2023-07-04T18:52:20.487506+0000","last_active":"2023-07-04T18:52:21.470363+0000","last_peered":"2023-07-04T18:52:21.470363+0000","last_clean":"2023-07-04T18:52:21.470363+0000","last_became_active":"2023-07-04T18:52:20.487248+0000","last_became_peered":"2023-07-04T18:52:20.487248+0000","last_unstale":"2023-07-04T18:52:21.470363+0000","last_undegraded":"2023-07-04T18:52:21.470363+0000","last_fullsized":"2023-07-04T18:52:21.470363+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470386+0000","last_change":"2023-07-04T18:52:20.485291+0000","last_active":"2023-07-04T18:52:21.470386+0000","last_peered":"2023-07-04T18:52:21.470386+0000","last_clean":"2023-07-04T18:52:21.470386+0000","last_became_active":"2023-07-04T18:52:20.484107+0000","last_became_peered":"2023-07-04T18:52:20.484107+0000","last_unstale":"2023-07-04T18:52:21.470386+0000","last_undegraded":"2023-07-04T18:52:21.470386+0000","last_fullsized":"2023-07-04T18:52:21.470386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:44.397113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064060+0000","last_change":"2023-07-04T18:52:20.492813+0000","last_active":"2023-07-04T18:52:22.064060+0000","last_peered":"2023-07-04T18:52:22.064060+0000","last_clean":"2023-07-04T18:52:22.064060+0000","last_became_active":"2023-07-04T18:52:20.492538+0000","last_became_peered":"2023-07-04T18:52:20.492538+0000","last_unstale":"2023-07-04T18:52:22.064060+0000","last_undegraded":"2023-07-04T18:52:22.064060+0000","last_fullsized":"2023-07-04T18:52:22.064060+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581360+0000","last_change":"2023-07-04T18:52:20.478667+0000","last_active":"2023-07-04T18:52:21.581360+0000","last_peered":"2023-07-04T18:52:21.581360+0000","last_clean":"2023-07-04T18:52:21.581360+0000","last_became_active":"2023-07-04T18:52:20.476017+0000","last_became_peered":"2023-07-04T18:52:20.476017+0000","last_unstale":"2023-07-04T18:52:21.581360+0000","last_undegraded":"2023-07-04T18:52:21.581360+0000","last_fullsized":"2023-07-04T18:52:21.581360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581190+0000","last_change":"2023-07-04T18:52:20.475096+0000","last_active":"2023-07-04T18:52:21.581190+0000","last_peered":"2023-07-04T18:52:21.581190+0000","last_clean":"2023-07-04T18:52:21.581190+0000","last_became_active":"2023-07-04T18:52:20.474728+0000","last_became_peered":"2023-07-04T18:52:20.474728+0000","last_unstale":"2023-07-04T18:52:21.581190+0000","last_undegraded":"2023-07-04T18:52:21.581190+0000","last_fullsized":"2023-07-04T18:52:21.581190+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:06:00.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063731+0000","last_change":"2023-07-04T18:52:20.479913+0000","last_active":"2023-07-04T18:52:22.063731+0000","last_peered":"2023-07-04T18:52:22.063731+0000","last_clean":"2023-07-04T18:52:22.063731+0000","last_became_active":"2023-07-04T18:52:20.479649+0000","last_became_peered":"2023-07-04T18:52:20.479649+0000","last_unstale":"2023-07-04T18:52:22.063731+0000","last_undegraded":"2023-07-04T18:52:22.063731+0000","last_fullsized":"2023-07-04T18:52:22.063731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:10.324552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583342+0000","last_change":"2023-07-04T18:52:20.494079+0000","last_active":"2023-07-04T18:52:21.583342+0000","last_peered":"2023-07-04T18:52:21.583342+0000","last_clean":"2023-07-04T18:52:21.583342+0000","last_became_active":"2023-07-04T18:52:20.493552+0000","last_became_peered":"2023-07-04T18:52:20.493552+0000","last_unstale":"2023-07-04T18:52:21.583342+0000","last_undegraded":"2023-07-04T18:52:21.583342+0000","last_fullsized":"2023-07-04T18:52:21.583342+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:30.610249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470099+0000","last_change":"2023-07-04T18:52:20.496203+0000","last_active":"2023-07-04T18:52:21.470099+0000","last_peered":"2023-07-04T18:52:21.470099+0000","last_clean":"2023-07-04T18:52:21.470099+0000","last_became_active":"2023-07-04T18:52:20.495950+0000","last_became_peered":"2023-07-04T18:52:20.495950+0000","last_unstale":"2023-07-04T18:52:21.470099+0000","last_undegraded":"2023-07-04T18:52:21.470099+0000","last_fullsized":"2023-07-04T18:52:21.470099+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064288+0000","last_change":"2023-07-04T18:52:20.497663+0000","last_active":"2023-07-04T18:52:22.064288+0000","last_peered":"2023-07-04T18:52:22.064288+0000","last_clean":"2023-07-04T18:52:22.064288+0000","last_became_active":"2023-07-04T18:52:20.497366+0000","last_became_peered":"2023-07-04T18:52:20.497366+0000","last_unstale":"2023-07-04T18:52:22.064288+0000","last_undegraded":"2023-07-04T18:52:22.064288+0000","last_fullsized":"2023-07-04T18:52:22.064288+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583734+0000","last_change":"2023-07-04T18:52:20.494517+0000","last_active":"2023-07-04T18:52:21.583734+0000","last_peered":"2023-07-04T18:52:21.583734+0000","last_clean":"2023-07-04T18:52:21.583734+0000","last_became_active":"2023-07-04T18:52:20.493002+0000","last_became_peered":"2023-07-04T18:52:20.493002+0000","last_unstale":"2023-07-04T18:52:21.583734+0000","last_undegraded":"2023-07-04T18:52:21.583734+0000","last_fullsized":"2023-07-04T18:52:21.583734+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583902+0000","last_change":"2023-07-04T18:52:20.492866+0000","last_active":"2023-07-04T18:52:21.583902+0000","last_peered":"2023-07-04T18:52:21.583902+0000","last_clean":"2023-07-04T18:52:21.583902+0000","last_became_active":"2023-07-04T18:52:20.490309+0000","last_became_peered":"2023-07-04T18:52:20.490309+0000","last_unstale":"2023-07-04T18:52:21.583902+0000","last_undegraded":"2023-07-04T18:52:21.583902+0000","last_fullsized":"2023-07-04T18:52:21.583902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:57.519386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064222+0000","last_change":"2023-07-04T18:52:20.480329+0000","last_active":"2023-07-04T18:52:22.064222+0000","last_peered":"2023-07-04T18:52:22.064222+0000","last_clean":"2023-07-04T18:52:22.064222+0000","last_became_active":"2023-07-04T18:52:20.480036+0000","last_became_peered":"2023-07-04T18:52:20.480036+0000","last_unstale":"2023-07-04T18:52:22.064222+0000","last_undegraded":"2023-07-04T18:52:22.064222+0000","last_fullsized":"2023-07-04T18:52:22.064222+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583547+0000","last_change":"2023-07-04T18:52:20.492993+0000","last_active":"2023-07-04T18:52:21.583547+0000","last_peered":"2023-07-04T18:52:21.583547+0000","last_clean":"2023-07-04T18:52:21.583547+0000","last_became_active":"2023-07-04T18:52:20.492523+0000","last_became_peered":"2023-07-04T18:52:20.492523+0000","last_unstale":"2023-07-04T18:52:21.583547+0000","last_undegraded":"2023-07-04T18:52:21.583547+0000","last_fullsized":"2023-07-04T18:52:21.583547+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:25.636206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471448+0000","last_change":"2023-07-04T18:52:20.476182+0000","last_active":"2023-07-04T18:52:21.471448+0000","last_peered":"2023-07-04T18:52:21.471448+0000","last_clean":"2023-07-04T18:52:21.471448+0000","last_became_active":"2023-07-04T18:52:20.475870+0000","last_became_peered":"2023-07-04T18:52:20.475870+0000","last_unstale":"2023-07-04T18:52:21.471448+0000","last_undegraded":"2023-07-04T18:52:21.471448+0000","last_fullsized":"2023-07-04T18:52:21.471448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583326+0000","last_change":"2023-07-04T18:52:20.493851+0000","last_active":"2023-07-04T18:52:21.583326+0000","last_peered":"2023-07-04T18:52:21.583326+0000","last_clean":"2023-07-04T18:52:21.583326+0000","last_became_active":"2023-07-04T18:52:20.493121+0000","last_became_peered":"2023-07-04T18:52:20.493121+0000","last_unstale":"2023-07-04T18:52:21.583326+0000","last_undegraded":"2023-07-04T18:52:21.583326+0000","last_fullsized":"2023-07-04T18:52:21.583326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064111+0000","last_change":"2023-07-04T18:52:20.496287+0000","last_active":"2023-07-04T18:52:22.064111+0000","last_peered":"2023-07-04T18:52:22.064111+0000","last_clean":"2023-07-04T18:52:22.064111+0000","last_became_active":"2023-07-04T18:52:20.495989+0000","last_became_peered":"2023-07-04T18:52:20.495989+0000","last_unstale":"2023-07-04T18:52:22.064111+0000","last_undegraded":"2023-07-04T18:52:22.064111+0000","last_fullsized":"2023-07-04T18:52:22.064111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064508+0000","last_change":"2023-07-04T18:52:20.497658+0000","last_active":"2023-07-04T18:52:22.064508+0000","last_peered":"2023-07-04T18:52:22.064508+0000","last_clean":"2023-07-04T18:52:22.064508+0000","last_became_active":"2023-07-04T18:52:20.497373+0000","last_became_peered":"2023-07-04T18:52:20.497373+0000","last_unstale":"2023-07-04T18:52:22.064508+0000","last_undegraded":"2023-07-04T18:52:22.064508+0000","last_fullsized":"2023-07-04T18:52:22.064508+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:12:06.480252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471789+0000","last_change":"2023-07-04T18:52:20.486609+0000","last_active":"2023-07-04T18:52:21.471789+0000","last_peered":"2023-07-04T18:52:21.471789+0000","last_clean":"2023-07-04T18:52:21.471789+0000","last_became_active":"2023-07-04T18:52:20.486292+0000","last_became_peered":"2023-07-04T18:52:20.486292+0000","last_unstale":"2023-07-04T18:52:21.471789+0000","last_undegraded":"2023-07-04T18:52:21.471789+0000","last_fullsized":"2023-07-04T18:52:21.471789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:44:15.915109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581263+0000","last_change":"2023-07-04T18:52:20.472502+0000","last_active":"2023-07-04T18:52:21.581263+0000","last_peered":"2023-07-04T18:52:21.581263+0000","last_clean":"2023-07-04T18:52:21.581263+0000","last_became_active":"2023-07-04T18:52:20.472250+0000","last_became_peered":"2023-07-04T18:52:20.472250+0000","last_unstale":"2023-07-04T18:52:21.581263+0000","last_undegraded":"2023-07-04T18:52:21.581263+0000","last_fullsized":"2023-07-04T18:52:21.581263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:04.134366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469450+0000","last_change":"2023-07-04T18:52:20.485739+0000","last_active":"2023-07-04T18:52:21.469450+0000","last_peered":"2023-07-04T18:52:21.469450+0000","last_clean":"2023-07-04T18:52:21.469450+0000","last_became_active":"2023-07-04T18:52:20.485405+0000","last_became_peered":"2023-07-04T18:52:20.485405+0000","last_unstale":"2023-07-04T18:52:21.469450+0000","last_undegraded":"2023-07-04T18:52:21.469450+0000","last_fullsized":"2023-07-04T18:52:21.469450+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583520+0000","last_change":"2023-07-04T18:52:20.488728+0000","last_active":"2023-07-04T18:52:21.583520+0000","last_peered":"2023-07-04T18:52:21.583520+0000","last_clean":"2023-07-04T18:52:21.583520+0000","last_became_active":"2023-07-04T18:52:20.488449+0000","last_became_peered":"2023-07-04T18:52:20.488449+0000","last_unstale":"2023-07-04T18:52:21.583520+0000","last_undegraded":"2023-07-04T18:52:21.583520+0000","last_fullsized":"2023-07-04T18:52:21.583520+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:04:28.623773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471182+0000","last_change":"2023-07-04T18:52:20.475095+0000","last_active":"2023-07-04T18:52:21.471182+0000","last_peered":"2023-07-04T18:52:21.471182+0000","last_clean":"2023-07-04T18:52:21.471182+0000","last_became_active":"2023-07-04T18:52:20.474780+0000","last_became_peered":"2023-07-04T18:52:20.474780+0000","last_unstale":"2023-07-04T18:52:21.471182+0000","last_undegraded":"2023-07-04T18:52:21.471182+0000","last_fullsized":"2023-07-04T18:52:21.471182+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064410+0000","last_change":"2023-07-04T18:52:20.478932+0000","last_active":"2023-07-04T18:52:22.064410+0000","last_peered":"2023-07-04T18:52:22.064410+0000","last_clean":"2023-07-04T18:52:22.064410+0000","last_became_active":"2023-07-04T18:52:20.477175+0000","last_became_peered":"2023-07-04T18:52:20.477175+0000","last_unstale":"2023-07-04T18:52:22.064410+0000","last_undegraded":"2023-07-04T18:52:22.064410+0000","last_fullsized":"2023-07-04T18:52:22.064410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:17.945865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581066+0000","last_change":"2023-07-04T18:52:20.476929+0000","last_active":"2023-07-04T18:52:21.581066+0000","last_peered":"2023-07-04T18:52:21.581066+0000","last_clean":"2023-07-04T18:52:21.581066+0000","last_became_active":"2023-07-04T18:52:20.476625+0000","last_became_peered":"2023-07-04T18:52:20.476625+0000","last_unstale":"2023-07-04T18:52:21.581066+0000","last_undegraded":"2023-07-04T18:52:21.581066+0000","last_fullsized":"2023-07-04T18:52:21.581066+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:00:15.725242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471820+0000","last_change":"2023-07-04T18:52:20.486108+0000","last_active":"2023-07-04T18:52:21.471820+0000","last_peered":"2023-07-04T18:52:21.471820+0000","last_clean":"2023-07-04T18:52:21.471820+0000","last_became_active":"2023-07-04T18:52:20.485841+0000","last_became_peered":"2023-07-04T18:52:20.485841+0000","last_unstale":"2023-07-04T18:52:21.471820+0000","last_undegraded":"2023-07-04T18:52:21.471820+0000","last_fullsized":"2023-07-04T18:52:21.471820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:23:46.685971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064525+0000","last_change":"2023-07-04T18:52:20.492895+0000","last_active":"2023-07-04T18:52:22.064525+0000","last_peered":"2023-07-04T18:52:22.064525+0000","last_clean":"2023-07-04T18:52:22.064525+0000","last_became_active":"2023-07-04T18:52:20.492622+0000","last_became_peered":"2023-07-04T18:52:20.492622+0000","last_unstale":"2023-07-04T18:52:22.064525+0000","last_undegraded":"2023-07-04T18:52:22.064525+0000","last_fullsized":"2023-07-04T18:52:22.064525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471649+0000","last_change":"2023-07-04T18:52:20.475874+0000","last_active":"2023-07-04T18:52:21.471649+0000","last_peered":"2023-07-04T18:52:21.471649+0000","last_clean":"2023-07-04T18:52:21.471649+0000","last_became_active":"2023-07-04T18:52:20.475558+0000","last_became_peered":"2023-07-04T18:52:20.475558+0000","last_unstale":"2023-07-04T18:52:21.471649+0000","last_undegraded":"2023-07-04T18:52:21.471649+0000","last_fullsized":"2023-07-04T18:52:21.471649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:51.752371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583963+0000","last_change":"2023-07-04T18:52:20.494100+0000","last_active":"2023-07-04T18:52:21.583963+0000","last_peered":"2023-07-04T18:52:21.583963+0000","last_clean":"2023-07-04T18:52:21.583963+0000","last_became_active":"2023-07-04T18:52:20.493125+0000","last_became_peered":"2023-07-04T18:52:20.493125+0000","last_unstale":"2023-07-04T18:52:21.583963+0000","last_undegraded":"2023-07-04T18:52:21.583963+0000","last_fullsized":"2023-07-04T18:52:21.583963+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469452+0000","last_change":"2023-07-04T18:52:20.477696+0000","last_active":"2023-07-04T18:52:21.469452+0000","last_peered":"2023-07-04T18:52:21.469452+0000","last_clean":"2023-07-04T18:52:21.469452+0000","last_became_active":"2023-07-04T18:52:20.477427+0000","last_became_peered":"2023-07-04T18:52:20.477427+0000","last_unstale":"2023-07-04T18:52:21.469452+0000","last_undegraded":"2023-07-04T18:52:21.469452+0000","last_fullsized":"2023-07-04T18:52:21.469452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469469+0000","last_change":"2023-07-04T18:52:20.479863+0000","last_active":"2023-07-04T18:52:21.469469+0000","last_peered":"2023-07-04T18:52:21.469469+0000","last_clean":"2023-07-04T18:52:21.469469+0000","last_became_active":"2023-07-04T18:52:20.479580+0000","last_became_peered":"2023-07-04T18:52:20.479580+0000","last_unstale":"2023-07-04T18:52:21.469469+0000","last_undegraded":"2023-07-04T18:52:21.469469+0000","last_fullsized":"2023-07-04T18:52:21.469469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064261+0000","last_change":"2023-07-04T18:52:20.478043+0000","last_active":"2023-07-04T18:52:22.064261+0000","last_peered":"2023-07-04T18:52:22.064261+0000","last_clean":"2023-07-04T18:52:22.064261+0000","last_became_active":"2023-07-04T18:52:20.476781+0000","last_became_peered":"2023-07-04T18:52:20.476781+0000","last_unstale":"2023-07-04T18:52:22.064261+0000","last_undegraded":"2023-07-04T18:52:22.064261+0000","last_fullsized":"2023-07-04T18:52:22.064261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063720+0000","last_change":"2023-07-04T18:52:20.472516+0000","last_active":"2023-07-04T18:52:22.063720+0000","last_peered":"2023-07-04T18:52:22.063720+0000","last_clean":"2023-07-04T18:52:22.063720+0000","last_became_active":"2023-07-04T18:52:20.471963+0000","last_became_peered":"2023-07-04T18:52:20.471963+0000","last_unstale":"2023-07-04T18:52:22.063720+0000","last_undegraded":"2023-07-04T18:52:22.063720+0000","last_fullsized":"2023-07-04T18:52:22.063720+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064356+0000","last_change":"2023-07-04T18:52:20.495269+0000","last_active":"2023-07-04T18:52:22.064356+0000","last_peered":"2023-07-04T18:52:22.064356+0000","last_clean":"2023-07-04T18:52:22.064356+0000","last_became_active":"2023-07-04T18:52:20.495024+0000","last_became_peered":"2023-07-04T18:52:20.495024+0000","last_unstale":"2023-07-04T18:52:22.064356+0000","last_undegraded":"2023-07-04T18:52:22.064356+0000","last_fullsized":"2023-07-04T18:52:22.064356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:47.935529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"36'4","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471842+0000","last_change":"2023-07-04T18:52:19.456618+0000","last_active":"2023-07-04T18:52:21.471842+0000","last_peered":"2023-07-04T18:52:21.471842+0000","last_clean":"2023-07-04T18:52:21.471842+0000","last_became_active":"2023-07-04T18:52:19.456387+0000","last_became_peered":"2023-07-04T18:52:19.456387+0000","last_unstale":"2023-07-04T18:52:21.471842+0000","last_undegraded":"2023-07-04T18:52:21.471842+0000","last_fullsized":"2023-07-04T18:52:21.471842+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:55:51.483834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064499+0000","last_change":"2023-07-04T18:52:20.498353+0000","last_active":"2023-07-04T18:52:22.064499+0000","last_peered":"2023-07-04T18:52:22.064499+0000","last_clean":"2023-07-04T18:52:22.064499+0000","last_became_active":"2023-07-04T18:52:20.498137+0000","last_became_peered":"2023-07-04T18:52:20.498137+0000","last_unstale":"2023-07-04T18:52:22.064499+0000","last_undegraded":"2023-07-04T18:52:22.064499+0000","last_fullsized":"2023-07-04T18:52:22.064499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:53:25.229985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064102+0000","last_change":"2023-07-04T18:52:19.462233+0000","last_active":"2023-07-04T18:52:22.064102+0000","last_peered":"2023-07-04T18:52:22.064102+0000","last_clean":"2023-07-04T18:52:22.064102+0000","last_became_active":"2023-07-04T18:52:19.461952+0000","last_became_peered":"2023-07-04T18:52:19.461952+0000","last_unstale":"2023-07-04T18:52:22.064102+0000","last_undegraded":"2023-07-04T18:52:22.064102+0000","last_fullsized":"2023-07-04T18:52:22.064102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:43.395483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471479+0000","last_change":"2023-07-04T18:52:20.475952+0000","last_active":"2023-07-04T18:52:21.471479+0000","last_peered":"2023-07-04T18:52:21.471479+0000","last_clean":"2023-07-04T18:52:21.471479+0000","last_became_active":"2023-07-04T18:52:20.475673+0000","last_became_peered":"2023-07-04T18:52:20.475673+0000","last_unstale":"2023-07-04T18:52:21.471479+0000","last_undegraded":"2023-07-04T18:52:21.471479+0000","last_fullsized":"2023-07-04T18:52:21.471479+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064496+0000","last_change":"2023-07-04T18:52:19.458951+0000","last_active":"2023-07-04T18:52:22.064496+0000","last_peered":"2023-07-04T18:52:22.064496+0000","last_clean":"2023-07-04T18:52:22.064496+0000","last_became_active":"2023-07-04T18:52:19.458681+0000","last_became_peered":"2023-07-04T18:52:19.458681+0000","last_unstale":"2023-07-04T18:52:22.064496+0000","last_undegraded":"2023-07-04T18:52:22.064496+0000","last_fullsized":"2023-07-04T18:52:22.064496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:29.608618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470459+0000","last_change":"2023-07-04T18:52:20.486218+0000","last_active":"2023-07-04T18:52:21.470459+0000","last_peered":"2023-07-04T18:52:21.470459+0000","last_clean":"2023-07-04T18:52:21.470459+0000","last_became_active":"2023-07-04T18:52:20.485895+0000","last_became_peered":"2023-07-04T18:52:20.485895+0000","last_unstale":"2023-07-04T18:52:21.470459+0000","last_undegraded":"2023-07-04T18:52:21.470459+0000","last_fullsized":"2023-07-04T18:52:21.470459+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471467+0000","last_change":"2023-07-04T18:52:19.466273+0000","last_active":"2023-07-04T18:52:21.471467+0000","last_peered":"2023-07-04T18:52:21.471467+0000","last_clean":"2023-07-04T18:52:21.471467+0000","last_became_active":"2023-07-04T18:52:19.465971+0000","last_became_peered":"2023-07-04T18:52:19.465971+0000","last_unstale":"2023-07-04T18:52:21.471467+0000","last_undegraded":"2023-07-04T18:52:21.471467+0000","last_fullsized":"2023-07-04T18:52:21.471467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063628+0000","last_change":"2023-07-04T18:52:20.485233+0000","last_active":"2023-07-04T18:52:22.063628+0000","last_peered":"2023-07-04T18:52:22.063628+0000","last_clean":"2023-07-04T18:52:22.063628+0000","last_became_active":"2023-07-04T18:52:20.484955+0000","last_became_peered":"2023-07-04T18:52:20.484955+0000","last_unstale":"2023-07-04T18:52:22.063628+0000","last_undegraded":"2023-07-04T18:52:22.063628+0000","last_fullsized":"2023-07-04T18:52:22.063628+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:46:36.992403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583679+0000","last_change":"2023-07-04T18:52:19.459600+0000","last_active":"2023-07-04T18:52:21.583679+0000","last_peered":"2023-07-04T18:52:21.583679+0000","last_clean":"2023-07-04T18:52:21.583679+0000","last_became_active":"2023-07-04T18:52:19.459350+0000","last_became_peered":"2023-07-04T18:52:19.459350+0000","last_unstale":"2023-07-04T18:52:21.583679+0000","last_undegraded":"2023-07-04T18:52:21.583679+0000","last_fullsized":"2023-07-04T18:52:21.583679+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:01.191883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583958+0000","last_change":"2023-07-04T18:52:20.492856+0000","last_active":"2023-07-04T18:52:21.583958+0000","last_peered":"2023-07-04T18:52:21.583958+0000","last_clean":"2023-07-04T18:52:21.583958+0000","last_became_active":"2023-07-04T18:52:20.492087+0000","last_became_peered":"2023-07-04T18:52:20.492087+0000","last_unstale":"2023-07-04T18:52:21.583958+0000","last_undegraded":"2023-07-04T18:52:21.583958+0000","last_fullsized":"2023-07-04T18:52:21.583958+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064204+0000","last_change":"2023-07-04T18:52:19.464499+0000","last_active":"2023-07-04T18:52:22.064204+0000","last_peered":"2023-07-04T18:52:22.064204+0000","last_clean":"2023-07-04T18:52:22.064204+0000","last_became_active":"2023-07-04T18:52:19.464217+0000","last_became_peered":"2023-07-04T18:52:19.464217+0000","last_unstale":"2023-07-04T18:52:22.064204+0000","last_undegraded":"2023-07-04T18:52:22.064204+0000","last_fullsized":"2023-07-04T18:52:22.064204+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471012+0000","last_change":"2023-07-04T18:52:20.494161+0000","last_active":"2023-07-04T18:52:21.471012+0000","last_peered":"2023-07-04T18:52:21.471012+0000","last_clean":"2023-07-04T18:52:21.471012+0000","last_became_active":"2023-07-04T18:52:20.493910+0000","last_became_peered":"2023-07-04T18:52:20.493910+0000","last_unstale":"2023-07-04T18:52:21.471012+0000","last_undegraded":"2023-07-04T18:52:21.471012+0000","last_fullsized":"2023-07-04T18:52:21.471012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063897+0000","last_change":"2023-07-04T18:52:19.461981+0000","last_active":"2023-07-04T18:52:22.063897+0000","last_peered":"2023-07-04T18:52:22.063897+0000","last_clean":"2023-07-04T18:52:22.063897+0000","last_became_active":"2023-07-04T18:52:19.461696+0000","last_became_peered":"2023-07-04T18:52:19.461696+0000","last_unstale":"2023-07-04T18:52:22.063897+0000","last_undegraded":"2023-07-04T18:52:22.063897+0000","last_fullsized":"2023-07-04T18:52:22.063897+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:04:28.702186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470561+0000","last_change":"2023-07-04T18:52:20.481579+0000","last_active":"2023-07-04T18:52:21.470561+0000","last_peered":"2023-07-04T18:52:21.470561+0000","last_clean":"2023-07-04T18:52:21.470561+0000","last_became_active":"2023-07-04T18:52:20.480982+0000","last_became_peered":"2023-07-04T18:52:20.480982+0000","last_unstale":"2023-07-04T18:52:21.470561+0000","last_undegraded":"2023-07-04T18:52:21.470561+0000","last_fullsized":"2023-07-04T18:52:21.470561+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470811+0000","last_change":"2023-07-04T18:52:19.455607+0000","last_active":"2023-07-04T18:52:21.470811+0000","last_peered":"2023-07-04T18:52:21.470811+0000","last_clean":"2023-07-04T18:52:21.470811+0000","last_became_active":"2023-07-04T18:52:19.455315+0000","last_became_peered":"2023-07-04T18:52:19.455315+0000","last_unstale":"2023-07-04T18:52:21.470811+0000","last_undegraded":"2023-07-04T18:52:21.470811+0000","last_fullsized":"2023-07-04T18:52:21.470811+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:24.634575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581368+0000","last_change":"2023-07-04T18:52:20.476740+0000","last_active":"2023-07-04T18:52:21.581368+0000","last_peered":"2023-07-04T18:52:21.581368+0000","last_clean":"2023-07-04T18:52:21.581368+0000","last_became_active":"2023-07-04T18:52:20.475692+0000","last_became_peered":"2023-07-04T18:52:20.475692+0000","last_unstale":"2023-07-04T18:52:21.581368+0000","last_undegraded":"2023-07-04T18:52:21.581368+0000","last_fullsized":"2023-07-04T18:52:21.581368+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470439+0000","last_change":"2023-07-04T18:52:19.466605+0000","last_active":"2023-07-04T18:52:21.470439+0000","last_peered":"2023-07-04T18:52:21.470439+0000","last_clean":"2023-07-04T18:52:21.470439+0000","last_became_active":"2023-07-04T18:52:19.466312+0000","last_became_peered":"2023-07-04T18:52:19.466312+0000","last_unstale":"2023-07-04T18:52:21.470439+0000","last_undegraded":"2023-07-04T18:52:21.470439+0000","last_fullsized":"2023-07-04T18:52:21.470439+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:46:12.053717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064462+0000","last_change":"2023-07-04T18:52:20.480353+0000","last_active":"2023-07-04T18:52:22.064462+0000","last_peered":"2023-07-04T18:52:22.064462+0000","last_clean":"2023-07-04T18:52:22.064462+0000","last_became_active":"2023-07-04T18:52:20.480033+0000","last_became_peered":"2023-07-04T18:52:20.480033+0000","last_unstale":"2023-07-04T18:52:22.064462+0000","last_undegraded":"2023-07-04T18:52:22.064462+0000","last_fullsized":"2023-07-04T18:52:22.064462+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583873+0000","last_change":"2023-07-04T18:52:19.466235+0000","last_active":"2023-07-04T18:52:21.583873+0000","last_peered":"2023-07-04T18:52:21.583873+0000","last_clean":"2023-07-04T18:52:21.583873+0000","last_became_active":"2023-07-04T18:52:19.465766+0000","last_became_peered":"2023-07-04T18:52:19.465766+0000","last_unstale":"2023-07-04T18:52:21.583873+0000","last_undegraded":"2023-07-04T18:52:21.583873+0000","last_fullsized":"2023-07-04T18:52:21.583873+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063868+0000","last_change":"2023-07-04T18:52:20.485459+0000","last_active":"2023-07-04T18:52:22.063868+0000","last_peered":"2023-07-04T18:52:22.063868+0000","last_clean":"2023-07-04T18:52:22.063868+0000","last_became_active":"2023-07-04T18:52:20.485176+0000","last_became_peered":"2023-07-04T18:52:20.485176+0000","last_unstale":"2023-07-04T18:52:22.063868+0000","last_undegraded":"2023-07-04T18:52:22.063868+0000","last_fullsized":"2023-07-04T18:52:22.063868+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"36'3","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583965+0000","last_change":"2023-07-04T18:52:19.458524+0000","last_active":"2023-07-04T18:52:21.583965+0000","last_peered":"2023-07-04T18:52:21.583965+0000","last_clean":"2023-07-04T18:52:21.583965+0000","last_became_active":"2023-07-04T18:52:19.458189+0000","last_became_peered":"2023-07-04T18:52:19.458189+0000","last_unstale":"2023-07-04T18:52:21.583965+0000","last_undegraded":"2023-07-04T18:52:21.583965+0000","last_fullsized":"2023-07-04T18:52:21.583965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:16.944234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471751+0000","last_change":"2023-07-04T18:52:20.473599+0000","last_active":"2023-07-04T18:52:21.471751+0000","last_peered":"2023-07-04T18:52:21.471751+0000","last_clean":"2023-07-04T18:52:21.471751+0000","last_became_active":"2023-07-04T18:52:20.473309+0000","last_became_peered":"2023-07-04T18:52:20.473309+0000","last_unstale":"2023-07-04T18:52:21.471751+0000","last_undegraded":"2023-07-04T18:52:21.471751+0000","last_fullsized":"2023-07-04T18:52:21.471751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"36'6","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470526+0000","last_change":"2023-07-04T18:52:19.465448+0000","last_active":"2023-07-04T18:52:21.470526+0000","last_peered":"2023-07-04T18:52:21.470526+0000","last_clean":"2023-07-04T18:52:21.470526+0000","last_became_active":"2023-07-04T18:52:19.465166+0000","last_became_peered":"2023-07-04T18:52:19.465166+0000","last_unstale":"2023-07-04T18:52:21.470526+0000","last_undegraded":"2023-07-04T18:52:21.470526+0000","last_fullsized":"2023-07-04T18:52:21.470526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:39.388420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063926+0000","last_change":"2023-07-04T18:52:20.496158+0000","last_active":"2023-07-04T18:52:22.063926+0000","last_peered":"2023-07-04T18:52:22.063926+0000","last_clean":"2023-07-04T18:52:22.063926+0000","last_became_active":"2023-07-04T18:52:20.495901+0000","last_became_peered":"2023-07-04T18:52:20.495901+0000","last_unstale":"2023-07-04T18:52:22.063926+0000","last_undegraded":"2023-07-04T18:52:22.063926+0000","last_fullsized":"2023-07-04T18:52:22.063926+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:09.892279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470569+0000","last_change":"2023-07-04T18:52:19.466047+0000","last_active":"2023-07-04T18:52:21.470569+0000","last_peered":"2023-07-04T18:52:21.470569+0000","last_clean":"2023-07-04T18:52:21.470569+0000","last_became_active":"2023-07-04T18:52:19.465676+0000","last_became_peered":"2023-07-04T18:52:19.465676+0000","last_unstale":"2023-07-04T18:52:21.470569+0000","last_undegraded":"2023-07-04T18:52:21.470569+0000","last_fullsized":"2023-07-04T18:52:21.470569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:09.322921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470149+0000","last_change":"2023-07-04T18:52:20.486940+0000","last_active":"2023-07-04T18:52:21.470149+0000","last_peered":"2023-07-04T18:52:21.470149+0000","last_clean":"2023-07-04T18:52:21.470149+0000","last_became_active":"2023-07-04T18:52:20.486728+0000","last_became_peered":"2023-07-04T18:52:20.486728+0000","last_unstale":"2023-07-04T18:52:21.470149+0000","last_undegraded":"2023-07-04T18:52:21.470149+0000","last_fullsized":"2023-07-04T18:52:21.470149+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063999+0000","last_change":"2023-07-04T18:52:19.458036+0000","last_active":"2023-07-04T18:52:22.063999+0000","last_peered":"2023-07-04T18:52:22.063999+0000","last_clean":"2023-07-04T18:52:22.063999+0000","last_became_active":"2023-07-04T18:52:19.457785+0000","last_became_peered":"2023-07-04T18:52:19.457785+0000","last_unstale":"2023-07-04T18:52:22.063999+0000","last_undegraded":"2023-07-04T18:52:22.063999+0000","last_fullsized":"2023-07-04T18:52:22.063999+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:30:57.141214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064474+0000","last_change":"2023-07-04T18:52:20.479886+0000","last_active":"2023-07-04T18:52:22.064474+0000","last_peered":"2023-07-04T18:52:22.064474+0000","last_clean":"2023-07-04T18:52:22.064474+0000","last_became_active":"2023-07-04T18:52:20.479590+0000","last_became_peered":"2023-07-04T18:52:20.479590+0000","last_unstale":"2023-07-04T18:52:22.064474+0000","last_undegraded":"2023-07-04T18:52:22.064474+0000","last_fullsized":"2023-07-04T18:52:22.064474+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:15.554299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469340+0000","last_change":"2023-07-04T18:52:19.456613+0000","last_active":"2023-07-04T18:52:21.469340+0000","last_peered":"2023-07-04T18:52:21.469340+0000","last_clean":"2023-07-04T18:52:21.469340+0000","last_became_active":"2023-07-04T18:52:19.456323+0000","last_became_peered":"2023-07-04T18:52:19.456323+0000","last_unstale":"2023-07-04T18:52:21.469340+0000","last_undegraded":"2023-07-04T18:52:21.469340+0000","last_fullsized":"2023-07-04T18:52:21.469340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:03.132735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470266+0000","last_change":"2023-07-04T18:52:20.486461+0000","last_active":"2023-07-04T18:52:21.470266+0000","last_peered":"2023-07-04T18:52:21.470266+0000","last_clean":"2023-07-04T18:52:21.470266+0000","last_became_active":"2023-07-04T18:52:20.486131+0000","last_became_peered":"2023-07-04T18:52:20.486131+0000","last_unstale":"2023-07-04T18:52:21.470266+0000","last_undegraded":"2023-07-04T18:52:21.470266+0000","last_fullsized":"2023-07-04T18:52:21.470266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:01.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"36'8","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064559+0000","last_change":"2023-07-04T18:52:19.464608+0000","last_active":"2023-07-04T18:52:22.064559+0000","last_peered":"2023-07-04T18:52:22.064559+0000","last_clean":"2023-07-04T18:52:22.064559+0000","last_became_active":"2023-07-04T18:52:19.464317+0000","last_became_peered":"2023-07-04T18:52:19.464317+0000","last_unstale":"2023-07-04T18:52:22.064559+0000","last_undegraded":"2023-07-04T18:52:22.064559+0000","last_fullsized":"2023-07-04T18:52:22.064559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:52.218872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470488+0000","last_change":"2023-07-04T18:52:20.484585+0000","last_active":"2023-07-04T18:52:21.470488+0000","last_peered":"2023-07-04T18:52:21.470488+0000","last_clean":"2023-07-04T18:52:21.470488+0000","last_became_active":"2023-07-04T18:52:20.484314+0000","last_became_peered":"2023-07-04T18:52:20.484314+0000","last_unstale":"2023-07-04T18:52:21.470488+0000","last_undegraded":"2023-07-04T18:52:21.470488+0000","last_fullsized":"2023-07-04T18:52:21.470488+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:06:27.571180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583814+0000","last_change":"2023-07-04T18:52:19.458147+0000","last_active":"2023-07-04T18:52:21.583814+0000","last_peered":"2023-07-04T18:52:21.583814+0000","last_clean":"2023-07-04T18:52:21.583814+0000","last_became_active":"2023-07-04T18:52:19.457690+0000","last_became_peered":"2023-07-04T18:52:19.457690+0000","last_unstale":"2023-07-04T18:52:21.583814+0000","last_undegraded":"2023-07-04T18:52:21.583814+0000","last_fullsized":"2023-07-04T18:52:21.583814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:10.064186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583832+0000","last_change":"2023-07-04T18:52:20.492732+0000","last_active":"2023-07-04T18:52:21.583832+0000","last_peered":"2023-07-04T18:52:21.583832+0000","last_clean":"2023-07-04T18:52:21.583832+0000","last_became_active":"2023-07-04T18:52:20.490197+0000","last_became_peered":"2023-07-04T18:52:20.490197+0000","last_unstale":"2023-07-04T18:52:21.583832+0000","last_undegraded":"2023-07-04T18:52:21.583832+0000","last_fullsized":"2023-07-04T18:52:21.583832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471601+0000","last_change":"2023-07-04T18:52:19.455963+0000","last_active":"2023-07-04T18:52:21.471601+0000","last_peered":"2023-07-04T18:52:21.471601+0000","last_clean":"2023-07-04T18:52:21.471601+0000","last_became_active":"2023-07-04T18:52:19.455693+0000","last_became_peered":"2023-07-04T18:52:19.455693+0000","last_unstale":"2023-07-04T18:52:21.471601+0000","last_undegraded":"2023-07-04T18:52:21.471601+0000","last_fullsized":"2023-07-04T18:52:21.471601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471710+0000","last_change":"2023-07-04T18:52:20.479233+0000","last_active":"2023-07-04T18:52:21.471710+0000","last_peered":"2023-07-04T18:52:21.471710+0000","last_clean":"2023-07-04T18:52:21.471710+0000","last_became_active":"2023-07-04T18:52:20.478983+0000","last_became_peered":"2023-07-04T18:52:20.478983+0000","last_unstale":"2023-07-04T18:52:21.471710+0000","last_undegraded":"2023-07-04T18:52:21.471710+0000","last_fullsized":"2023-07-04T18:52:21.471710+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:28:52.986168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581163+0000","last_change":"2023-07-04T18:52:19.462190+0000","last_active":"2023-07-04T18:52:21.581163+0000","last_peered":"2023-07-04T18:52:21.581163+0000","last_clean":"2023-07-04T18:52:21.581163+0000","last_became_active":"2023-07-04T18:52:19.461906+0000","last_became_peered":"2023-07-04T18:52:19.461906+0000","last_unstale":"2023-07-04T18:52:21.581163+0000","last_undegraded":"2023-07-04T18:52:21.581163+0000","last_fullsized":"2023-07-04T18:52:21.581163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:48.049300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581307+0000","last_change":"2023-07-04T18:52:20.479758+0000","last_active":"2023-07-04T18:52:21.581307+0000","last_peered":"2023-07-04T18:52:21.581307+0000","last_clean":"2023-07-04T18:52:21.581307+0000","last_became_active":"2023-07-04T18:52:20.479319+0000","last_became_peered":"2023-07-04T18:52:20.479319+0000","last_unstale":"2023-07-04T18:52:21.581307+0000","last_undegraded":"2023-07-04T18:52:21.581307+0000","last_fullsized":"2023-07-04T18:52:21.581307+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063995+0000","last_change":"2023-07-04T18:52:19.457634+0000","last_active":"2023-07-04T18:52:22.063995+0000","last_peered":"2023-07-04T18:52:22.063995+0000","last_clean":"2023-07-04T18:52:22.063995+0000","last_became_active":"2023-07-04T18:52:19.457341+0000","last_became_peered":"2023-07-04T18:52:19.457341+0000","last_unstale":"2023-07-04T18:52:22.063995+0000","last_undegraded":"2023-07-04T18:52:22.063995+0000","last_fullsized":"2023-07-04T18:52:22.063995+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:35:03.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064181+0000","last_change":"2023-07-04T18:52:20.483273+0000","last_active":"2023-07-04T18:52:22.064181+0000","last_peered":"2023-07-04T18:52:22.064181+0000","last_clean":"2023-07-04T18:52:22.064181+0000","last_became_active":"2023-07-04T18:52:20.480069+0000","last_became_peered":"2023-07-04T18:52:20.480069+0000","last_unstale":"2023-07-04T18:52:22.064181+0000","last_undegraded":"2023-07-04T18:52:22.064181+0000","last_fullsized":"2023-07-04T18:52:22.064181+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581171+0000","last_change":"2023-07-04T18:52:19.458392+0000","last_active":"2023-07-04T18:52:21.581171+0000","last_peered":"2023-07-04T18:52:21.581171+0000","last_clean":"2023-07-04T18:52:21.581171+0000","last_became_active":"2023-07-04T18:52:19.458046+0000","last_became_peered":"2023-07-04T18:52:19.458046+0000","last_unstale":"2023-07-04T18:52:21.581171+0000","last_undegraded":"2023-07-04T18:52:21.581171+0000","last_fullsized":"2023-07-04T18:52:21.581171+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:40:12.555311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063694+0000","last_change":"2023-07-04T18:52:20.482831+0000","last_active":"2023-07-04T18:52:22.063694+0000","last_peered":"2023-07-04T18:52:22.063694+0000","last_clean":"2023-07-04T18:52:22.063694+0000","last_became_active":"2023-07-04T18:52:20.482494+0000","last_became_peered":"2023-07-04T18:52:20.482494+0000","last_unstale":"2023-07-04T18:52:22.063694+0000","last_undegraded":"2023-07-04T18:52:22.063694+0000","last_fullsized":"2023-07-04T18:52:22.063694+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:45:42.643724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581048+0000","last_change":"2023-07-04T18:52:19.456664+0000","last_active":"2023-07-04T18:52:21.581048+0000","last_peered":"2023-07-04T18:52:21.581048+0000","last_clean":"2023-07-04T18:52:21.581048+0000","last_became_active":"2023-07-04T18:52:19.456374+0000","last_became_peered":"2023-07-04T18:52:19.456374+0000","last_unstale":"2023-07-04T18:52:21.581048+0000","last_undegraded":"2023-07-04T18:52:21.581048+0000","last_fullsized":"2023-07-04T18:52:21.581048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:04:05.770013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583383+0000","last_change":"2023-07-04T18:52:20.496556+0000","last_active":"2023-07-04T18:52:21.583383+0000","last_peered":"2023-07-04T18:52:21.583383+0000","last_clean":"2023-07-04T18:52:21.583383+0000","last_became_active":"2023-07-04T18:52:20.495594+0000","last_became_peered":"2023-07-04T18:52:20.495594+0000","last_unstale":"2023-07-04T18:52:21.583383+0000","last_undegraded":"2023-07-04T18:52:21.583383+0000","last_fullsized":"2023-07-04T18:52:21.583383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:29:33.232347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064681+0000","last_change":"2023-07-04T18:52:19.466675+0000","last_active":"2023-07-04T18:52:22.064681+0000","last_peered":"2023-07-04T18:52:22.064681+0000","last_clean":"2023-07-04T18:52:22.064681+0000","last_became_active":"2023-07-04T18:52:19.466465+0000","last_became_peered":"2023-07-04T18:52:19.466465+0000","last_unstale":"2023-07-04T18:52:22.064681+0000","last_undegraded":"2023-07-04T18:52:22.064681+0000","last_fullsized":"2023-07-04T18:52:22.064681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:38:13.573589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470404+0000","last_change":"2023-07-04T18:52:20.481606+0000","last_active":"2023-07-04T18:52:21.470404+0000","last_peered":"2023-07-04T18:52:21.470404+0000","last_clean":"2023-07-04T18:52:21.470404+0000","last_became_active":"2023-07-04T18:52:20.481341+0000","last_became_peered":"2023-07-04T18:52:20.481341+0000","last_unstale":"2023-07-04T18:52:21.470404+0000","last_undegraded":"2023-07-04T18:52:21.470404+0000","last_fullsized":"2023-07-04T18:52:21.470404+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:58.532607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581050+0000","last_change":"2023-07-04T18:52:19.457999+0000","last_active":"2023-07-04T18:52:21.581050+0000","last_peered":"2023-07-04T18:52:21.581050+0000","last_clean":"2023-07-04T18:52:21.581050+0000","last_became_active":"2023-07-04T18:52:19.457744+0000","last_became_peered":"2023-07-04T18:52:19.457744+0000","last_unstale":"2023-07-04T18:52:21.581050+0000","last_undegraded":"2023-07-04T18:52:21.581050+0000","last_fullsized":"2023-07-04T18:52:21.581050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:08:10.842309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063778+0000","last_change":"2023-07-04T18:52:20.485157+0000","last_active":"2023-07-04T18:52:22.063778+0000","last_peered":"2023-07-04T18:52:22.063778+0000","last_clean":"2023-07-04T18:52:22.063778+0000","last_became_active":"2023-07-04T18:52:20.484906+0000","last_became_peered":"2023-07-04T18:52:20.484906+0000","last_unstale":"2023-07-04T18:52:22.063778+0000","last_undegraded":"2023-07-04T18:52:22.063778+0000","last_fullsized":"2023-07-04T18:52:22.063778+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064086+0000","last_change":"2023-07-04T18:52:19.461058+0000","last_active":"2023-07-04T18:52:22.064086+0000","last_peered":"2023-07-04T18:52:22.064086+0000","last_clean":"2023-07-04T18:52:22.064086+0000","last_became_active":"2023-07-04T18:52:19.460786+0000","last_became_peered":"2023-07-04T18:52:19.460786+0000","last_unstale":"2023-07-04T18:52:22.064086+0000","last_undegraded":"2023-07-04T18:52:22.064086+0000","last_fullsized":"2023-07-04T18:52:22.064086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:57.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064059+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:52:22.064059+0000","last_peered":"2023-07-04T18:52:22.064059+0000","last_clean":"2023-07-04T18:52:22.064059+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:52:22.064059+0000","last_undegraded":"2023-07-04T18:52:22.064059+0000","last_fullsized":"2023-07-04T18:52:22.064059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:45:06.368879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470452+0000","last_change":"2023-07-04T18:52:20.491037+0000","last_active":"2023-07-04T18:52:21.470452+0000","last_peered":"2023-07-04T18:52:21.470452+0000","last_clean":"2023-07-04T18:52:21.470452+0000","last_became_active":"2023-07-04T18:52:20.490723+0000","last_became_peered":"2023-07-04T18:52:20.490723+0000","last_unstale":"2023-07-04T18:52:21.470452+0000","last_undegraded":"2023-07-04T18:52:21.470452+0000","last_fullsized":"2023-07-04T18:52:21.470452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063944+0000","last_change":"2023-07-04T18:52:19.462323+0000","last_active":"2023-07-04T18:52:22.063944+0000","last_peered":"2023-07-04T18:52:22.063944+0000","last_clean":"2023-07-04T18:52:22.063944+0000","last_became_active":"2023-07-04T18:52:19.462057+0000","last_became_peered":"2023-07-04T18:52:19.462057+0000","last_unstale":"2023-07-04T18:52:22.063944+0000","last_undegraded":"2023-07-04T18:52:22.063944+0000","last_fullsized":"2023-07-04T18:52:22.063944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:00.356116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583488+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:52:21.583488+0000","last_peered":"2023-07-04T18:52:21.583488+0000","last_clean":"2023-07-04T18:52:21.583488+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:52:21.583488+0000","last_undegraded":"2023-07-04T18:52:21.583488+0000","last_fullsized":"2023-07-04T18:52:21.583488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:21:22.798814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064397+0000","last_change":"2023-07-04T18:52:20.484156+0000","last_active":"2023-07-04T18:52:22.064397+0000","last_peered":"2023-07-04T18:52:22.064397+0000","last_clean":"2023-07-04T18:52:22.064397+0000","last_became_active":"2023-07-04T18:52:20.483878+0000","last_became_peered":"2023-07-04T18:52:20.483878+0000","last_unstale":"2023-07-04T18:52:22.064397+0000","last_undegraded":"2023-07-04T18:52:22.064397+0000","last_fullsized":"2023-07-04T18:52:22.064397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581165+0000","last_change":"2023-07-04T18:52:19.460102+0000","last_active":"2023-07-04T18:52:21.581165+0000","last_peered":"2023-07-04T18:52:21.581165+0000","last_clean":"2023-07-04T18:52:21.581165+0000","last_became_active":"2023-07-04T18:52:19.459830+0000","last_became_peered":"2023-07-04T18:52:19.459830+0000","last_unstale":"2023-07-04T18:52:21.581165+0000","last_undegraded":"2023-07-04T18:52:21.581165+0000","last_fullsized":"2023-07-04T18:52:21.581165+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:17:54.321667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581016+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:52:21.581016+0000","last_peered":"2023-07-04T18:52:21.581016+0000","last_clean":"2023-07-04T18:52:21.581016+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:52:21.581016+0000","last_undegraded":"2023-07-04T18:52:21.581016+0000","last_fullsized":"2023-07-04T18:52:21.581016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:13:46.966327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583836+0000","last_change":"2023-07-04T18:52:20.489978+0000","last_active":"2023-07-04T18:52:21.583836+0000","last_peered":"2023-07-04T18:52:21.583836+0000","last_clean":"2023-07-04T18:52:21.583836+0000","last_became_active":"2023-07-04T18:52:20.489690+0000","last_became_peered":"2023-07-04T18:52:20.489690+0000","last_unstale":"2023-07-04T18:52:21.583836+0000","last_undegraded":"2023-07-04T18:52:21.583836+0000","last_fullsized":"2023-07-04T18:52:21.583836+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064479+0000","last_change":"2023-07-04T18:52:19.469850+0000","last_active":"2023-07-04T18:52:22.064479+0000","last_peered":"2023-07-04T18:52:22.064479+0000","last_clean":"2023-07-04T18:52:22.064479+0000","last_became_active":"2023-07-04T18:52:19.469546+0000","last_became_peered":"2023-07-04T18:52:19.469546+0000","last_unstale":"2023-07-04T18:52:22.064479+0000","last_undegraded":"2023-07-04T18:52:22.064479+0000","last_fullsized":"2023-07-04T18:52:22.064479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:28:10.562186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583631+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:52:21.583631+0000","last_peered":"2023-07-04T18:52:21.583631+0000","last_clean":"2023-07-04T18:52:21.583631+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:52:21.583631+0000","last_undegraded":"2023-07-04T18:52:21.583631+0000","last_fullsized":"2023-07-04T18:52:21.583631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:49.853323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":197,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064304+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:52:22.064304+0000","last_peered":"2023-07-04T18:52:22.064304+0000","last_clean":"2023-07-04T18:52:22.064304+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:52:22.064304+0000","last_undegraded":"2023-07-04T18:52:22.064304+0000","last_fullsized":"2023-07-04T18:52:22.064304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:43.685172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063939+0000","last_change":"2023-07-04T18:52:20.497660+0000","last_active":"2023-07-04T18:52:22.063939+0000","last_peered":"2023-07-04T18:52:22.063939+0000","last_clean":"2023-07-04T18:52:22.063939+0000","last_became_active":"2023-07-04T18:52:20.497337+0000","last_became_peered":"2023-07-04T18:52:20.497337+0000","last_unstale":"2023-07-04T18:52:22.063939+0000","last_undegraded":"2023-07-04T18:52:22.063939+0000","last_fullsized":"2023-07-04T18:52:22.063939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:20:18.389953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583996+0000","last_change":"2023-07-04T18:52:19.461997+0000","last_active":"2023-07-04T18:52:21.583996+0000","last_peered":"2023-07-04T18:52:21.583996+0000","last_clean":"2023-07-04T18:52:21.583996+0000","last_became_active":"2023-07-04T18:52:19.461815+0000","last_became_peered":"2023-07-04T18:52:19.461815+0000","last_unstale":"2023-07-04T18:52:21.583996+0000","last_undegraded":"2023-07-04T18:52:21.583996+0000","last_fullsized":"2023-07-04T18:52:21.583996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:14.552668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064565+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:52:22.064565+0000","last_peered":"2023-07-04T18:52:22.064565+0000","last_clean":"2023-07-04T18:52:22.064565+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:52:22.064565+0000","last_undegraded":"2023-07-04T18:52:22.064565+0000","last_fullsized":"2023-07-04T18:52:22.064565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:01:20.555239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063962+0000","last_change":"2023-07-04T18:52:20.477535+0000","last_active":"2023-07-04T18:52:22.063962+0000","last_peered":"2023-07-04T18:52:22.063962+0000","last_clean":"2023-07-04T18:52:22.063962+0000","last_became_active":"2023-07-04T18:52:20.477247+0000","last_became_peered":"2023-07-04T18:52:20.477247+0000","last_unstale":"2023-07-04T18:52:22.063962+0000","last_undegraded":"2023-07-04T18:52:22.063962+0000","last_fullsized":"2023-07-04T18:52:22.063962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:00:07.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469531+0000","last_change":"2023-07-04T18:52:19.455490+0000","last_active":"2023-07-04T18:52:21.469531+0000","last_peered":"2023-07-04T18:52:21.469531+0000","last_clean":"2023-07-04T18:52:21.469531+0000","last_became_active":"2023-07-04T18:52:19.455224+0000","last_became_peered":"2023-07-04T18:52:19.455224+0000","last_unstale":"2023-07-04T18:52:21.469531+0000","last_undegraded":"2023-07-04T18:52:21.469531+0000","last_fullsized":"2023-07-04T18:52:21.469531+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:53:35.991529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064762+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:52:22.064762+0000","last_peered":"2023-07-04T18:52:22.064762+0000","last_clean":"2023-07-04T18:52:22.064762+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:52:22.064762+0000","last_undegraded":"2023-07-04T18:52:22.064762+0000","last_fullsized":"2023-07-04T18:52:22.064762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:26:23.252619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471172+0000","last_change":"2023-07-04T18:52:20.487224+0000","last_active":"2023-07-04T18:52:21.471172+0000","last_peered":"2023-07-04T18:52:21.471172+0000","last_clean":"2023-07-04T18:52:21.471172+0000","last_became_active":"2023-07-04T18:52:20.486638+0000","last_became_peered":"2023-07-04T18:52:20.486638+0000","last_unstale":"2023-07-04T18:52:21.471172+0000","last_undegraded":"2023-07-04T18:52:21.471172+0000","last_fullsized":"2023-07-04T18:52:21.471172+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:02.193514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470860+0000","last_change":"2023-07-04T18:52:19.465554+0000","last_active":"2023-07-04T18:52:21.470860+0000","last_peered":"2023-07-04T18:52:21.470860+0000","last_clean":"2023-07-04T18:52:21.470860+0000","last_became_active":"2023-07-04T18:52:19.465268+0000","last_became_peered":"2023-07-04T18:52:19.465268+0000","last_unstale":"2023-07-04T18:52:21.470860+0000","last_undegraded":"2023-07-04T18:52:21.470860+0000","last_fullsized":"2023-07-04T18:52:21.470860+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:22:24.930028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064543+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:52:22.064543+0000","last_peered":"2023-07-04T18:52:22.064543+0000","last_clean":"2023-07-04T18:52:22.064543+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:52:22.064543+0000","last_undegraded":"2023-07-04T18:52:22.064543+0000","last_fullsized":"2023-07-04T18:52:22.064543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:28:30.099502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064698+0000","last_change":"2023-07-04T18:52:20.479835+0000","last_active":"2023-07-04T18:52:22.064698+0000","last_peered":"2023-07-04T18:52:22.064698+0000","last_clean":"2023-07-04T18:52:22.064698+0000","last_became_active":"2023-07-04T18:52:20.479561+0000","last_became_peered":"2023-07-04T18:52:20.479561+0000","last_unstale":"2023-07-04T18:52:22.064698+0000","last_undegraded":"2023-07-04T18:52:22.064698+0000","last_fullsized":"2023-07-04T18:52:22.064698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:59:27.432907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470121+0000","last_change":"2023-07-04T18:52:19.461537+0000","last_active":"2023-07-04T18:52:21.470121+0000","last_peered":"2023-07-04T18:52:21.470121+0000","last_clean":"2023-07-04T18:52:21.470121+0000","last_became_active":"2023-07-04T18:52:19.461279+0000","last_became_peered":"2023-07-04T18:52:19.461279+0000","last_unstale":"2023-07-04T18:52:21.470121+0000","last_undegraded":"2023-07-04T18:52:21.470121+0000","last_fullsized":"2023-07-04T18:52:21.470121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:08.890649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471908+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:52:21.471908+0000","last_peered":"2023-07-04T18:52:21.471908+0000","last_clean":"2023-07-04T18:52:21.471908+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:52:21.471908+0000","last_undegraded":"2023-07-04T18:52:21.471908+0000","last_fullsized":"2023-07-04T18:52:21.471908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:11:33.840191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":1032192,"data_stored":767390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.66000000000000003}]},{"osd":1,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":0.68300000000000005}]},{"osd":2,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.55100000000000005}]},{"osd":3,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.61799999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.497},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.59699999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63}]},{"osd":6,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53200000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54700000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.64600000000000002}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.57999999999999996}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.434},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.51200000000000001}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.27200000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.371}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":3.0600000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.626}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.68300000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77600000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":1019904,"data_stored":765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.441}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.7719999999999998,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.751},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69699999999999995}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":3.7109999999999999,"5min":3.7109999999999999,"15min":3.7109999999999999},"last":0.80800000000000005}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.79600000000000004}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.72099999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.371},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.313}]},{"osd":1,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.35899999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.34499999999999997}]},{"osd":4,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.505},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.436}]},{"osd":5,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.49099999999999999}]},{"osd":6,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48099999999999998}]},{"osd":7,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":438272,"data_stored":176986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.315}]},{"osd":1,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27600000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.54200000000000004}]},{"osd":5,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.65600000000000003}]},{"osd":6,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.748},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70599999999999996}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.65100000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.84999999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.82699999999999996}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.77700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.50700000000000001}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.78700000000000003}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.85399999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.97799999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.95799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:52:29.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-04T18:52:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.263+0000 7efdc97c9700 1 Processor -- start 2023-07-04T18:52:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.263+0000 7efdc97c9700 1 -- start start 2023-07-04T18:52:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc97c9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc40b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc97c9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efdc40af8f0 0x7efdc40b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc97c9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc40b2790 con 0x7efdc40b1860 2023-07-04T18:52:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc97c9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc40b2a70 con 0x7efdc40af8f0 2023-07-04T18:52:29.270 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc97c9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40b2d50 con 0x7efdc40d33c0 2023-07-04T18:52:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc2ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efdc40b1860 0x7efdc40d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53186/0 (socket says 172.21.15.77:53186) 2023-07-04T18:52:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc2ffd700 1 -- 172.21.15.77:0/586107815 learned_addr learned my addr 172.21.15.77:0/586107815 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 --2- 172.21.15.77:0/586107815 >> v2:172.21.15.77:3300/0 conn(0x7efdc40af8f0 0x7efdc40b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc27fc700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc40b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 -- 172.21.15.77:0/586107815 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 msgr2=0x7efdc40b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:29.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc40b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 -- 172.21.15.77:0/586107815 >> v1:172.21.15.77:6789/0 conn(0x7efdc40b1860 legacy=0x7efdc40d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 -- 172.21.15.77:0/586107815 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc40b3030 con 0x7efdc40af8f0 2023-07-04T18:52:29.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc1ffb700 1 -- 172.21.15.77:0/586107815 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233580471 0 0) 0x7efdc40b2790 con 0x7efdc40b1860 2023-07-04T18:52:29.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc37fe700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40af8f0 0x7efdc40b3d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009d90 tx=0x7efdbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf221a0bb36f01c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.264+0000 7efdc1ffb700 1 -- 172.21.15.77:0/586107815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00e070 con 0x7efdc40af8f0 2023-07-04T18:52:29.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc1ffb700 1 -- 172.21.15.77:0/586107815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc00bbd0 con 0x7efdc40af8f0 2023-07-04T18:52:29.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc1ffb700 1 -- 172.21.15.77:0/586107815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc012430 con 0x7efdc40af8f0 2023-07-04T18:52:29.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 -- 172.21.15.77:0/586107815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40af8f0 msgr2=0x7efdc40b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40af8f0 0x7efdc40b3d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009d90 tx=0x7efdbc00b330 comp rx=0 tx=0).stop 2023-07-04T18:52:29.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 -- 172.21.15.77:0/586107815 shutdown_connections 2023-07-04T18:52:29.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40af8f0 0x7efdc40b3d70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 --2- 172.21.15.77:0/586107815 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.274 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 -- 172.21.15.77:0/586107815 >> 172.21.15.77:0/586107815 conn(0x7efdc4071e90 msgr2=0x7efdc409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 -- 172.21.15.77:0/586107815 shutdown_connections 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.265+0000 7efdc97c9700 1 -- 172.21.15.77:0/586107815 wait complete. 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.266+0000 7efdc97c9700 1 Processor -- start 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.266+0000 7efdc97c9700 1 -- start start 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 0x7efdc415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.275 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc415b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40c9a20 con 0x7efdc40b1860 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415b6d0 con 0x7efdc40af8f0 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc97c9700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40c9460 con 0x7efdc40d33c0 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46054/0 (socket says 172.21.15.77:46054) 2023-07-04T18:52:29.276 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc27fc700 1 -- 172.21.15.77:0/2420828179 learned_addr learned my addr 172.21.15.77:0/2420828179 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc2ffd700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 0x7efdc415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.267+0000 7efdc37fe700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc415b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 -- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 msgr2=0x7efdc415b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc415b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 -- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 msgr2=0x7efdc415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.277 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 0x7efdc415b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 -- 172.21.15.77:0/2420828179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc009a70 con 0x7efdc40b1860 2023-07-04T18:52:29.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc2ffd700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 0x7efdc415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:29.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc37fe700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc415b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:29.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc27fc700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efdb400d800 tx=0x7efdb400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.278 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40125f0 con 0x7efdc40b1860 2023-07-04T18:52:29.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb4012750 con 0x7efdc40b1860 2023-07-04T18:52:29.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc40c9740 con 0x7efdc40b1860 2023-07-04T18:52:29.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4012b00 con 0x7efdc40b1860 2023-07-04T18:52:29.279 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.268+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc40c7750 con 0x7efdc40b1860 2023-07-04T18:52:29.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.269+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc40c72d0 con 0x7efdc40b1860 2023-07-04T18:52:29.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.269+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efdb4012c60 con 0x7efdc40b1860 2023-07-04T18:52:29.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.270+0000 7efdab7fe700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 0x7efdb006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.281 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.270+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7efdb4089850 con 0x7efdc40b1860 2023-07-04T18:52:29.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.270+0000 7efdc2ffd700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 0x7efdb006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.270+0000 7efdc2ffd700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 0x7efdb006d190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000f60 tx=0x7efdb8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.282 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.274+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efdc40c72d0 con 0x7efdc40b1860 2023-07-04T18:52:29.460 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.459+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 --> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efdc40b3310 con 0x7efdb006ace0 2023-07-04T18:52:29.470 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.468+0000 7efdab7fe700 1 -- 172.21.15.77:0/2420828179 <== mgr.4100 v2:172.21.15.82:6832/3872286367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311359 (secure 0 0 0) 0x7efdb8001ca0 con 0x7efdb006ace0 2023-07-04T18:52:29.474 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:29.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 msgr2=0x7efdb006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 0x7efdb006d190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000f60 tx=0x7efdb8009040 comp rx=0 tx=0).stop 2023-07-04T18:52:29.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 msgr2=0x7efdc40c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.476 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efdb400d800 tx=0x7efdb400dbc0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 shutdown_connections 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7efdb006ace0 0x7efdb006d190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7efdc40d33c0 0x7efdc415b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdc40af8f0 0x7efdc415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 --2- 172.21.15.77:0/2420828179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efdc40b1860 0x7efdc40c7c70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 >> 172.21.15.77:0/2420828179 conn(0x7efdc4071e90 msgr2=0x7efdc40b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:29.477 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 shutdown_connections 2023-07-04T18:52:29.478 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.471+0000 7efdc97c9700 1 -- 172.21.15.77:0/2420828179 wait complete. 2023-07-04T18:52:29.478 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-07-04T18:52:29.486 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2023-07-04T18:52:29.193255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220136,"kb_used_data":4520,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754754584,"statfs":{"total":773094113280,"available":772868694016,"internally_reserved":0,"allocated":4628480,"data_stored":2588564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.726348"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470363+0000","last_change":"2023-07-04T18:52:20.487506+0000","last_active":"2023-07-04T18:52:21.470363+0000","last_peered":"2023-07-04T18:52:21.470363+0000","last_clean":"2023-07-04T18:52:21.470363+0000","last_became_active":"2023-07-04T18:52:20.487248+0000","last_became_peered":"2023-07-04T18:52:20.487248+0000","last_unstale":"2023-07-04T18:52:21.470363+0000","last_undegraded":"2023-07-04T18:52:21.470363+0000","last_fullsized":"2023-07-04T18:52:21.470363+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470386+0000","last_change":"2023-07-04T18:52:20.485291+0000","last_active":"2023-07-04T18:52:21.470386+0000","last_peered":"2023-07-04T18:52:21.470386+0000","last_clean":"2023-07-04T18:52:21.470386+0000","last_became_active":"2023-07-04T18:52:20.484107+0000","last_became_peered":"2023-07-04T18:52:20.484107+0000","last_unstale":"2023-07-04T18:52:21.470386+0000","last_undegraded":"2023-07-04T18:52:21.470386+0000","last_fullsized":"2023-07-04T18:52:21.470386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:44.397113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064060+0000","last_change":"2023-07-04T18:52:20.492813+0000","last_active":"2023-07-04T18:52:22.064060+0000","last_peered":"2023-07-04T18:52:22.064060+0000","last_clean":"2023-07-04T18:52:22.064060+0000","last_became_active":"2023-07-04T18:52:20.492538+0000","last_became_peered":"2023-07-04T18:52:20.492538+0000","last_unstale":"2023-07-04T18:52:22.064060+0000","last_undegraded":"2023-07-04T18:52:22.064060+0000","last_fullsized":"2023-07-04T18:52:22.064060+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:58:51.005756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581360+0000","last_change":"2023-07-04T18:52:20.478667+0000","last_active":"2023-07-04T18:52:21.581360+0000","last_peered":"2023-07-04T18:52:21.581360+0000","last_clean":"2023-07-04T18:52:21.581360+0000","last_became_active":"2023-07-04T18:52:20.476017+0000","last_became_peered":"2023-07-04T18:52:20.476017+0000","last_unstale":"2023-07-04T18:52:21.581360+0000","last_undegraded":"2023-07-04T18:52:21.581360+0000","last_fullsized":"2023-07-04T18:52:21.581360+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581190+0000","last_change":"2023-07-04T18:52:20.475096+0000","last_active":"2023-07-04T18:52:21.581190+0000","last_peered":"2023-07-04T18:52:21.581190+0000","last_clean":"2023-07-04T18:52:21.581190+0000","last_became_active":"2023-07-04T18:52:20.474728+0000","last_became_peered":"2023-07-04T18:52:20.474728+0000","last_unstale":"2023-07-04T18:52:21.581190+0000","last_undegraded":"2023-07-04T18:52:21.581190+0000","last_fullsized":"2023-07-04T18:52:21.581190+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:06:00.352618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063731+0000","last_change":"2023-07-04T18:52:20.479913+0000","last_active":"2023-07-04T18:52:22.063731+0000","last_peered":"2023-07-04T18:52:22.063731+0000","last_clean":"2023-07-04T18:52:22.063731+0000","last_became_active":"2023-07-04T18:52:20.479649+0000","last_became_peered":"2023-07-04T18:52:20.479649+0000","last_unstale":"2023-07-04T18:52:22.063731+0000","last_undegraded":"2023-07-04T18:52:22.063731+0000","last_fullsized":"2023-07-04T18:52:22.063731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:10.324552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583342+0000","last_change":"2023-07-04T18:52:20.494079+0000","last_active":"2023-07-04T18:52:21.583342+0000","last_peered":"2023-07-04T18:52:21.583342+0000","last_clean":"2023-07-04T18:52:21.583342+0000","last_became_active":"2023-07-04T18:52:20.493552+0000","last_became_peered":"2023-07-04T18:52:20.493552+0000","last_unstale":"2023-07-04T18:52:21.583342+0000","last_undegraded":"2023-07-04T18:52:21.583342+0000","last_fullsized":"2023-07-04T18:52:21.583342+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:30.610249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470099+0000","last_change":"2023-07-04T18:52:20.496203+0000","last_active":"2023-07-04T18:52:21.470099+0000","last_peered":"2023-07-04T18:52:21.470099+0000","last_clean":"2023-07-04T18:52:21.470099+0000","last_became_active":"2023-07-04T18:52:20.495950+0000","last_became_peered":"2023-07-04T18:52:20.495950+0000","last_unstale":"2023-07-04T18:52:21.470099+0000","last_undegraded":"2023-07-04T18:52:21.470099+0000","last_fullsized":"2023-07-04T18:52:21.470099+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064288+0000","last_change":"2023-07-04T18:52:20.497663+0000","last_active":"2023-07-04T18:52:22.064288+0000","last_peered":"2023-07-04T18:52:22.064288+0000","last_clean":"2023-07-04T18:52:22.064288+0000","last_became_active":"2023-07-04T18:52:20.497366+0000","last_became_peered":"2023-07-04T18:52:20.497366+0000","last_unstale":"2023-07-04T18:52:22.064288+0000","last_undegraded":"2023-07-04T18:52:22.064288+0000","last_fullsized":"2023-07-04T18:52:22.064288+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583734+0000","last_change":"2023-07-04T18:52:20.494517+0000","last_active":"2023-07-04T18:52:21.583734+0000","last_peered":"2023-07-04T18:52:21.583734+0000","last_clean":"2023-07-04T18:52:21.583734+0000","last_became_active":"2023-07-04T18:52:20.493002+0000","last_became_peered":"2023-07-04T18:52:20.493002+0000","last_unstale":"2023-07-04T18:52:21.583734+0000","last_undegraded":"2023-07-04T18:52:21.583734+0000","last_fullsized":"2023-07-04T18:52:21.583734+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583902+0000","last_change":"2023-07-04T18:52:20.492866+0000","last_active":"2023-07-04T18:52:21.583902+0000","last_peered":"2023-07-04T18:52:21.583902+0000","last_clean":"2023-07-04T18:52:21.583902+0000","last_became_active":"2023-07-04T18:52:20.490309+0000","last_became_peered":"2023-07-04T18:52:20.490309+0000","last_unstale":"2023-07-04T18:52:21.583902+0000","last_undegraded":"2023-07-04T18:52:21.583902+0000","last_fullsized":"2023-07-04T18:52:21.583902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:57.519386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064222+0000","last_change":"2023-07-04T18:52:20.480329+0000","last_active":"2023-07-04T18:52:22.064222+0000","last_peered":"2023-07-04T18:52:22.064222+0000","last_clean":"2023-07-04T18:52:22.064222+0000","last_became_active":"2023-07-04T18:52:20.480036+0000","last_became_peered":"2023-07-04T18:52:20.480036+0000","last_unstale":"2023-07-04T18:52:22.064222+0000","last_undegraded":"2023-07-04T18:52:22.064222+0000","last_fullsized":"2023-07-04T18:52:22.064222+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583547+0000","last_change":"2023-07-04T18:52:20.492993+0000","last_active":"2023-07-04T18:52:21.583547+0000","last_peered":"2023-07-04T18:52:21.583547+0000","last_clean":"2023-07-04T18:52:21.583547+0000","last_became_active":"2023-07-04T18:52:20.492523+0000","last_became_peered":"2023-07-04T18:52:20.492523+0000","last_unstale":"2023-07-04T18:52:21.583547+0000","last_undegraded":"2023-07-04T18:52:21.583547+0000","last_fullsized":"2023-07-04T18:52:21.583547+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:25.636206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471448+0000","last_change":"2023-07-04T18:52:20.476182+0000","last_active":"2023-07-04T18:52:21.471448+0000","last_peered":"2023-07-04T18:52:21.471448+0000","last_clean":"2023-07-04T18:52:21.471448+0000","last_became_active":"2023-07-04T18:52:20.475870+0000","last_became_peered":"2023-07-04T18:52:20.475870+0000","last_unstale":"2023-07-04T18:52:21.471448+0000","last_undegraded":"2023-07-04T18:52:21.471448+0000","last_fullsized":"2023-07-04T18:52:21.471448+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583326+0000","last_change":"2023-07-04T18:52:20.493851+0000","last_active":"2023-07-04T18:52:21.583326+0000","last_peered":"2023-07-04T18:52:21.583326+0000","last_clean":"2023-07-04T18:52:21.583326+0000","last_became_active":"2023-07-04T18:52:20.493121+0000","last_became_peered":"2023-07-04T18:52:20.493121+0000","last_unstale":"2023-07-04T18:52:21.583326+0000","last_undegraded":"2023-07-04T18:52:21.583326+0000","last_fullsized":"2023-07-04T18:52:21.583326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064111+0000","last_change":"2023-07-04T18:52:20.496287+0000","last_active":"2023-07-04T18:52:22.064111+0000","last_peered":"2023-07-04T18:52:22.064111+0000","last_clean":"2023-07-04T18:52:22.064111+0000","last_became_active":"2023-07-04T18:52:20.495989+0000","last_became_peered":"2023-07-04T18:52:20.495989+0000","last_unstale":"2023-07-04T18:52:22.064111+0000","last_undegraded":"2023-07-04T18:52:22.064111+0000","last_fullsized":"2023-07-04T18:52:22.064111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064508+0000","last_change":"2023-07-04T18:52:20.497658+0000","last_active":"2023-07-04T18:52:22.064508+0000","last_peered":"2023-07-04T18:52:22.064508+0000","last_clean":"2023-07-04T18:52:22.064508+0000","last_became_active":"2023-07-04T18:52:20.497373+0000","last_became_peered":"2023-07-04T18:52:20.497373+0000","last_unstale":"2023-07-04T18:52:22.064508+0000","last_undegraded":"2023-07-04T18:52:22.064508+0000","last_fullsized":"2023-07-04T18:52:22.064508+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:12:06.480252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471789+0000","last_change":"2023-07-04T18:52:20.486609+0000","last_active":"2023-07-04T18:52:21.471789+0000","last_peered":"2023-07-04T18:52:21.471789+0000","last_clean":"2023-07-04T18:52:21.471789+0000","last_became_active":"2023-07-04T18:52:20.486292+0000","last_became_peered":"2023-07-04T18:52:20.486292+0000","last_unstale":"2023-07-04T18:52:21.471789+0000","last_undegraded":"2023-07-04T18:52:21.471789+0000","last_fullsized":"2023-07-04T18:52:21.471789+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:44:15.915109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581263+0000","last_change":"2023-07-04T18:52:20.472502+0000","last_active":"2023-07-04T18:52:21.581263+0000","last_peered":"2023-07-04T18:52:21.581263+0000","last_clean":"2023-07-04T18:52:21.581263+0000","last_became_active":"2023-07-04T18:52:20.472250+0000","last_became_peered":"2023-07-04T18:52:20.472250+0000","last_unstale":"2023-07-04T18:52:21.581263+0000","last_undegraded":"2023-07-04T18:52:21.581263+0000","last_fullsized":"2023-07-04T18:52:21.581263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:04.134366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469450+0000","last_change":"2023-07-04T18:52:20.485739+0000","last_active":"2023-07-04T18:52:21.469450+0000","last_peered":"2023-07-04T18:52:21.469450+0000","last_clean":"2023-07-04T18:52:21.469450+0000","last_became_active":"2023-07-04T18:52:20.485405+0000","last_became_peered":"2023-07-04T18:52:20.485405+0000","last_unstale":"2023-07-04T18:52:21.469450+0000","last_undegraded":"2023-07-04T18:52:21.469450+0000","last_fullsized":"2023-07-04T18:52:21.469450+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583520+0000","last_change":"2023-07-04T18:52:20.488728+0000","last_active":"2023-07-04T18:52:21.583520+0000","last_peered":"2023-07-04T18:52:21.583520+0000","last_clean":"2023-07-04T18:52:21.583520+0000","last_became_active":"2023-07-04T18:52:20.488449+0000","last_became_peered":"2023-07-04T18:52:20.488449+0000","last_unstale":"2023-07-04T18:52:21.583520+0000","last_undegraded":"2023-07-04T18:52:21.583520+0000","last_fullsized":"2023-07-04T18:52:21.583520+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:04:28.623773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471182+0000","last_change":"2023-07-04T18:52:20.475095+0000","last_active":"2023-07-04T18:52:21.471182+0000","last_peered":"2023-07-04T18:52:21.471182+0000","last_clean":"2023-07-04T18:52:21.471182+0000","last_became_active":"2023-07-04T18:52:20.474780+0000","last_became_peered":"2023-07-04T18:52:20.474780+0000","last_unstale":"2023-07-04T18:52:21.471182+0000","last_undegraded":"2023-07-04T18:52:21.471182+0000","last_fullsized":"2023-07-04T18:52:21.471182+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064410+0000","last_change":"2023-07-04T18:52:20.478932+0000","last_active":"2023-07-04T18:52:22.064410+0000","last_peered":"2023-07-04T18:52:22.064410+0000","last_clean":"2023-07-04T18:52:22.064410+0000","last_became_active":"2023-07-04T18:52:20.477175+0000","last_became_peered":"2023-07-04T18:52:20.477175+0000","last_unstale":"2023-07-04T18:52:22.064410+0000","last_undegraded":"2023-07-04T18:52:22.064410+0000","last_fullsized":"2023-07-04T18:52:22.064410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:17.945865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581066+0000","last_change":"2023-07-04T18:52:20.476929+0000","last_active":"2023-07-04T18:52:21.581066+0000","last_peered":"2023-07-04T18:52:21.581066+0000","last_clean":"2023-07-04T18:52:21.581066+0000","last_became_active":"2023-07-04T18:52:20.476625+0000","last_became_peered":"2023-07-04T18:52:20.476625+0000","last_unstale":"2023-07-04T18:52:21.581066+0000","last_undegraded":"2023-07-04T18:52:21.581066+0000","last_fullsized":"2023-07-04T18:52:21.581066+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:00:15.725242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471820+0000","last_change":"2023-07-04T18:52:20.486108+0000","last_active":"2023-07-04T18:52:21.471820+0000","last_peered":"2023-07-04T18:52:21.471820+0000","last_clean":"2023-07-04T18:52:21.471820+0000","last_became_active":"2023-07-04T18:52:20.485841+0000","last_became_peered":"2023-07-04T18:52:20.485841+0000","last_unstale":"2023-07-04T18:52:21.471820+0000","last_undegraded":"2023-07-04T18:52:21.471820+0000","last_fullsized":"2023-07-04T18:52:21.471820+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:23:46.685971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064525+0000","last_change":"2023-07-04T18:52:20.492895+0000","last_active":"2023-07-04T18:52:22.064525+0000","last_peered":"2023-07-04T18:52:22.064525+0000","last_clean":"2023-07-04T18:52:22.064525+0000","last_became_active":"2023-07-04T18:52:20.492622+0000","last_became_peered":"2023-07-04T18:52:20.492622+0000","last_unstale":"2023-07-04T18:52:22.064525+0000","last_undegraded":"2023-07-04T18:52:22.064525+0000","last_fullsized":"2023-07-04T18:52:22.064525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:48:04.255981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471649+0000","last_change":"2023-07-04T18:52:20.475874+0000","last_active":"2023-07-04T18:52:21.471649+0000","last_peered":"2023-07-04T18:52:21.471649+0000","last_clean":"2023-07-04T18:52:21.471649+0000","last_became_active":"2023-07-04T18:52:20.475558+0000","last_became_peered":"2023-07-04T18:52:20.475558+0000","last_unstale":"2023-07-04T18:52:21.471649+0000","last_undegraded":"2023-07-04T18:52:21.471649+0000","last_fullsized":"2023-07-04T18:52:21.471649+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:51.752371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583963+0000","last_change":"2023-07-04T18:52:20.494100+0000","last_active":"2023-07-04T18:52:21.583963+0000","last_peered":"2023-07-04T18:52:21.583963+0000","last_clean":"2023-07-04T18:52:21.583963+0000","last_became_active":"2023-07-04T18:52:20.493125+0000","last_became_peered":"2023-07-04T18:52:20.493125+0000","last_unstale":"2023-07-04T18:52:21.583963+0000","last_undegraded":"2023-07-04T18:52:21.583963+0000","last_fullsized":"2023-07-04T18:52:21.583963+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469452+0000","last_change":"2023-07-04T18:52:20.477696+0000","last_active":"2023-07-04T18:52:21.469452+0000","last_peered":"2023-07-04T18:52:21.469452+0000","last_clean":"2023-07-04T18:52:21.469452+0000","last_became_active":"2023-07-04T18:52:20.477427+0000","last_became_peered":"2023-07-04T18:52:20.477427+0000","last_unstale":"2023-07-04T18:52:21.469452+0000","last_undegraded":"2023-07-04T18:52:21.469452+0000","last_fullsized":"2023-07-04T18:52:21.469452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:32:00.968766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469469+0000","last_change":"2023-07-04T18:52:20.479863+0000","last_active":"2023-07-04T18:52:21.469469+0000","last_peered":"2023-07-04T18:52:21.469469+0000","last_clean":"2023-07-04T18:52:21.469469+0000","last_became_active":"2023-07-04T18:52:20.479580+0000","last_became_peered":"2023-07-04T18:52:20.479580+0000","last_unstale":"2023-07-04T18:52:21.469469+0000","last_undegraded":"2023-07-04T18:52:21.469469+0000","last_fullsized":"2023-07-04T18:52:21.469469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064261+0000","last_change":"2023-07-04T18:52:20.478043+0000","last_active":"2023-07-04T18:52:22.064261+0000","last_peered":"2023-07-04T18:52:22.064261+0000","last_clean":"2023-07-04T18:52:22.064261+0000","last_became_active":"2023-07-04T18:52:20.476781+0000","last_became_peered":"2023-07-04T18:52:20.476781+0000","last_unstale":"2023-07-04T18:52:22.064261+0000","last_undegraded":"2023-07-04T18:52:22.064261+0000","last_fullsized":"2023-07-04T18:52:22.064261+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063720+0000","last_change":"2023-07-04T18:52:20.472516+0000","last_active":"2023-07-04T18:52:22.063720+0000","last_peered":"2023-07-04T18:52:22.063720+0000","last_clean":"2023-07-04T18:52:22.063720+0000","last_became_active":"2023-07-04T18:52:20.471963+0000","last_became_peered":"2023-07-04T18:52:20.471963+0000","last_unstale":"2023-07-04T18:52:22.063720+0000","last_undegraded":"2023-07-04T18:52:22.063720+0000","last_fullsized":"2023-07-04T18:52:22.063720+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064356+0000","last_change":"2023-07-04T18:52:20.495269+0000","last_active":"2023-07-04T18:52:22.064356+0000","last_peered":"2023-07-04T18:52:22.064356+0000","last_clean":"2023-07-04T18:52:22.064356+0000","last_became_active":"2023-07-04T18:52:20.495024+0000","last_became_peered":"2023-07-04T18:52:20.495024+0000","last_unstale":"2023-07-04T18:52:22.064356+0000","last_undegraded":"2023-07-04T18:52:22.064356+0000","last_fullsized":"2023-07-04T18:52:22.064356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:47.935529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"36'4","reported_seq":17,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471842+0000","last_change":"2023-07-04T18:52:19.456618+0000","last_active":"2023-07-04T18:52:21.471842+0000","last_peered":"2023-07-04T18:52:21.471842+0000","last_clean":"2023-07-04T18:52:21.471842+0000","last_became_active":"2023-07-04T18:52:19.456387+0000","last_became_peered":"2023-07-04T18:52:19.456387+0000","last_unstale":"2023-07-04T18:52:21.471842+0000","last_undegraded":"2023-07-04T18:52:21.471842+0000","last_fullsized":"2023-07-04T18:52:21.471842+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:55:51.483834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064499+0000","last_change":"2023-07-04T18:52:20.498353+0000","last_active":"2023-07-04T18:52:22.064499+0000","last_peered":"2023-07-04T18:52:22.064499+0000","last_clean":"2023-07-04T18:52:22.064499+0000","last_became_active":"2023-07-04T18:52:20.498137+0000","last_became_peered":"2023-07-04T18:52:20.498137+0000","last_unstale":"2023-07-04T18:52:22.064499+0000","last_undegraded":"2023-07-04T18:52:22.064499+0000","last_fullsized":"2023-07-04T18:52:22.064499+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:53:25.229985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064102+0000","last_change":"2023-07-04T18:52:19.462233+0000","last_active":"2023-07-04T18:52:22.064102+0000","last_peered":"2023-07-04T18:52:22.064102+0000","last_clean":"2023-07-04T18:52:22.064102+0000","last_became_active":"2023-07-04T18:52:19.461952+0000","last_became_peered":"2023-07-04T18:52:19.461952+0000","last_unstale":"2023-07-04T18:52:22.064102+0000","last_undegraded":"2023-07-04T18:52:22.064102+0000","last_fullsized":"2023-07-04T18:52:22.064102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:57:43.395483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471479+0000","last_change":"2023-07-04T18:52:20.475952+0000","last_active":"2023-07-04T18:52:21.471479+0000","last_peered":"2023-07-04T18:52:21.471479+0000","last_clean":"2023-07-04T18:52:21.471479+0000","last_became_active":"2023-07-04T18:52:20.475673+0000","last_became_peered":"2023-07-04T18:52:20.475673+0000","last_unstale":"2023-07-04T18:52:21.471479+0000","last_undegraded":"2023-07-04T18:52:21.471479+0000","last_fullsized":"2023-07-04T18:52:21.471479+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:11.065816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064496+0000","last_change":"2023-07-04T18:52:19.458951+0000","last_active":"2023-07-04T18:52:22.064496+0000","last_peered":"2023-07-04T18:52:22.064496+0000","last_clean":"2023-07-04T18:52:22.064496+0000","last_became_active":"2023-07-04T18:52:19.458681+0000","last_became_peered":"2023-07-04T18:52:19.458681+0000","last_unstale":"2023-07-04T18:52:22.064496+0000","last_undegraded":"2023-07-04T18:52:22.064496+0000","last_fullsized":"2023-07-04T18:52:22.064496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:59:29.608618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470459+0000","last_change":"2023-07-04T18:52:20.486218+0000","last_active":"2023-07-04T18:52:21.470459+0000","last_peered":"2023-07-04T18:52:21.470459+0000","last_clean":"2023-07-04T18:52:21.470459+0000","last_became_active":"2023-07-04T18:52:20.485895+0000","last_became_peered":"2023-07-04T18:52:20.485895+0000","last_unstale":"2023-07-04T18:52:21.470459+0000","last_undegraded":"2023-07-04T18:52:21.470459+0000","last_fullsized":"2023-07-04T18:52:21.470459+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:31:05.903989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471467+0000","last_change":"2023-07-04T18:52:19.466273+0000","last_active":"2023-07-04T18:52:21.471467+0000","last_peered":"2023-07-04T18:52:21.471467+0000","last_clean":"2023-07-04T18:52:21.471467+0000","last_became_active":"2023-07-04T18:52:19.465971+0000","last_became_peered":"2023-07-04T18:52:19.465971+0000","last_unstale":"2023-07-04T18:52:21.471467+0000","last_undegraded":"2023-07-04T18:52:21.471467+0000","last_fullsized":"2023-07-04T18:52:21.471467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063628+0000","last_change":"2023-07-04T18:52:20.485233+0000","last_active":"2023-07-04T18:52:22.063628+0000","last_peered":"2023-07-04T18:52:22.063628+0000","last_clean":"2023-07-04T18:52:22.063628+0000","last_became_active":"2023-07-04T18:52:20.484955+0000","last_became_peered":"2023-07-04T18:52:20.484955+0000","last_unstale":"2023-07-04T18:52:22.063628+0000","last_undegraded":"2023-07-04T18:52:22.063628+0000","last_fullsized":"2023-07-04T18:52:22.063628+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:46:36.992403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583679+0000","last_change":"2023-07-04T18:52:19.459600+0000","last_active":"2023-07-04T18:52:21.583679+0000","last_peered":"2023-07-04T18:52:21.583679+0000","last_clean":"2023-07-04T18:52:21.583679+0000","last_became_active":"2023-07-04T18:52:19.459350+0000","last_became_peered":"2023-07-04T18:52:19.459350+0000","last_unstale":"2023-07-04T18:52:21.583679+0000","last_undegraded":"2023-07-04T18:52:21.583679+0000","last_fullsized":"2023-07-04T18:52:21.583679+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:01.191883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583958+0000","last_change":"2023-07-04T18:52:20.492856+0000","last_active":"2023-07-04T18:52:21.583958+0000","last_peered":"2023-07-04T18:52:21.583958+0000","last_clean":"2023-07-04T18:52:21.583958+0000","last_became_active":"2023-07-04T18:52:20.492087+0000","last_became_peered":"2023-07-04T18:52:20.492087+0000","last_unstale":"2023-07-04T18:52:21.583958+0000","last_undegraded":"2023-07-04T18:52:21.583958+0000","last_fullsized":"2023-07-04T18:52:21.583958+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064204+0000","last_change":"2023-07-04T18:52:19.464499+0000","last_active":"2023-07-04T18:52:22.064204+0000","last_peered":"2023-07-04T18:52:22.064204+0000","last_clean":"2023-07-04T18:52:22.064204+0000","last_became_active":"2023-07-04T18:52:19.464217+0000","last_became_peered":"2023-07-04T18:52:19.464217+0000","last_unstale":"2023-07-04T18:52:22.064204+0000","last_undegraded":"2023-07-04T18:52:22.064204+0000","last_fullsized":"2023-07-04T18:52:22.064204+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:02:56.517755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471012+0000","last_change":"2023-07-04T18:52:20.494161+0000","last_active":"2023-07-04T18:52:21.471012+0000","last_peered":"2023-07-04T18:52:21.471012+0000","last_clean":"2023-07-04T18:52:21.471012+0000","last_became_active":"2023-07-04T18:52:20.493910+0000","last_became_peered":"2023-07-04T18:52:20.493910+0000","last_unstale":"2023-07-04T18:52:21.471012+0000","last_undegraded":"2023-07-04T18:52:21.471012+0000","last_fullsized":"2023-07-04T18:52:21.471012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:56:34.274338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063897+0000","last_change":"2023-07-04T18:52:19.461981+0000","last_active":"2023-07-04T18:52:22.063897+0000","last_peered":"2023-07-04T18:52:22.063897+0000","last_clean":"2023-07-04T18:52:22.063897+0000","last_became_active":"2023-07-04T18:52:19.461696+0000","last_became_peered":"2023-07-04T18:52:19.461696+0000","last_unstale":"2023-07-04T18:52:22.063897+0000","last_undegraded":"2023-07-04T18:52:22.063897+0000","last_fullsized":"2023-07-04T18:52:22.063897+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:04:28.702186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470561+0000","last_change":"2023-07-04T18:52:20.481579+0000","last_active":"2023-07-04T18:52:21.470561+0000","last_peered":"2023-07-04T18:52:21.470561+0000","last_clean":"2023-07-04T18:52:21.470561+0000","last_became_active":"2023-07-04T18:52:20.480982+0000","last_became_peered":"2023-07-04T18:52:20.480982+0000","last_unstale":"2023-07-04T18:52:21.470561+0000","last_undegraded":"2023-07-04T18:52:21.470561+0000","last_fullsized":"2023-07-04T18:52:21.470561+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:22:01.094083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470811+0000","last_change":"2023-07-04T18:52:19.455607+0000","last_active":"2023-07-04T18:52:21.470811+0000","last_peered":"2023-07-04T18:52:21.470811+0000","last_clean":"2023-07-04T18:52:21.470811+0000","last_became_active":"2023-07-04T18:52:19.455315+0000","last_became_peered":"2023-07-04T18:52:19.455315+0000","last_unstale":"2023-07-04T18:52:21.470811+0000","last_undegraded":"2023-07-04T18:52:21.470811+0000","last_fullsized":"2023-07-04T18:52:21.470811+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:42:24.634575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581368+0000","last_change":"2023-07-04T18:52:20.476740+0000","last_active":"2023-07-04T18:52:21.581368+0000","last_peered":"2023-07-04T18:52:21.581368+0000","last_clean":"2023-07-04T18:52:21.581368+0000","last_became_active":"2023-07-04T18:52:20.475692+0000","last_became_peered":"2023-07-04T18:52:20.475692+0000","last_unstale":"2023-07-04T18:52:21.581368+0000","last_undegraded":"2023-07-04T18:52:21.581368+0000","last_fullsized":"2023-07-04T18:52:21.581368+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:53:44.293071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470439+0000","last_change":"2023-07-04T18:52:19.466605+0000","last_active":"2023-07-04T18:52:21.470439+0000","last_peered":"2023-07-04T18:52:21.470439+0000","last_clean":"2023-07-04T18:52:21.470439+0000","last_became_active":"2023-07-04T18:52:19.466312+0000","last_became_peered":"2023-07-04T18:52:19.466312+0000","last_unstale":"2023-07-04T18:52:21.470439+0000","last_undegraded":"2023-07-04T18:52:21.470439+0000","last_fullsized":"2023-07-04T18:52:21.470439+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:46:12.053717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064462+0000","last_change":"2023-07-04T18:52:20.480353+0000","last_active":"2023-07-04T18:52:22.064462+0000","last_peered":"2023-07-04T18:52:22.064462+0000","last_clean":"2023-07-04T18:52:22.064462+0000","last_became_active":"2023-07-04T18:52:20.480033+0000","last_became_peered":"2023-07-04T18:52:20.480033+0000","last_unstale":"2023-07-04T18:52:22.064462+0000","last_undegraded":"2023-07-04T18:52:22.064462+0000","last_fullsized":"2023-07-04T18:52:22.064462+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583873+0000","last_change":"2023-07-04T18:52:19.466235+0000","last_active":"2023-07-04T18:52:21.583873+0000","last_peered":"2023-07-04T18:52:21.583873+0000","last_clean":"2023-07-04T18:52:21.583873+0000","last_became_active":"2023-07-04T18:52:19.465766+0000","last_became_peered":"2023-07-04T18:52:19.465766+0000","last_unstale":"2023-07-04T18:52:21.583873+0000","last_undegraded":"2023-07-04T18:52:21.583873+0000","last_fullsized":"2023-07-04T18:52:21.583873+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063868+0000","last_change":"2023-07-04T18:52:20.485459+0000","last_active":"2023-07-04T18:52:22.063868+0000","last_peered":"2023-07-04T18:52:22.063868+0000","last_clean":"2023-07-04T18:52:22.063868+0000","last_became_active":"2023-07-04T18:52:20.485176+0000","last_became_peered":"2023-07-04T18:52:20.485176+0000","last_unstale":"2023-07-04T18:52:22.063868+0000","last_undegraded":"2023-07-04T18:52:22.063868+0000","last_fullsized":"2023-07-04T18:52:22.063868+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"36'3","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583965+0000","last_change":"2023-07-04T18:52:19.458524+0000","last_active":"2023-07-04T18:52:21.583965+0000","last_peered":"2023-07-04T18:52:21.583965+0000","last_clean":"2023-07-04T18:52:21.583965+0000","last_became_active":"2023-07-04T18:52:19.458189+0000","last_became_peered":"2023-07-04T18:52:19.458189+0000","last_unstale":"2023-07-04T18:52:21.583965+0000","last_undegraded":"2023-07-04T18:52:21.583965+0000","last_fullsized":"2023-07-04T18:52:21.583965+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:34:16.944234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471751+0000","last_change":"2023-07-04T18:52:20.473599+0000","last_active":"2023-07-04T18:52:21.471751+0000","last_peered":"2023-07-04T18:52:21.471751+0000","last_clean":"2023-07-04T18:52:21.471751+0000","last_became_active":"2023-07-04T18:52:20.473309+0000","last_became_peered":"2023-07-04T18:52:20.473309+0000","last_unstale":"2023-07-04T18:52:21.471751+0000","last_undegraded":"2023-07-04T18:52:21.471751+0000","last_fullsized":"2023-07-04T18:52:21.471751+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:09:29.087726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"36'6","reported_seq":19,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470526+0000","last_change":"2023-07-04T18:52:19.465448+0000","last_active":"2023-07-04T18:52:21.470526+0000","last_peered":"2023-07-04T18:52:21.470526+0000","last_clean":"2023-07-04T18:52:21.470526+0000","last_became_active":"2023-07-04T18:52:19.465166+0000","last_became_peered":"2023-07-04T18:52:19.465166+0000","last_unstale":"2023-07-04T18:52:21.470526+0000","last_undegraded":"2023-07-04T18:52:21.470526+0000","last_fullsized":"2023-07-04T18:52:21.470526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:39:39.388420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063926+0000","last_change":"2023-07-04T18:52:20.496158+0000","last_active":"2023-07-04T18:52:22.063926+0000","last_peered":"2023-07-04T18:52:22.063926+0000","last_clean":"2023-07-04T18:52:22.063926+0000","last_became_active":"2023-07-04T18:52:20.495901+0000","last_became_peered":"2023-07-04T18:52:20.495901+0000","last_unstale":"2023-07-04T18:52:22.063926+0000","last_undegraded":"2023-07-04T18:52:22.063926+0000","last_fullsized":"2023-07-04T18:52:22.063926+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:09.892279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470569+0000","last_change":"2023-07-04T18:52:19.466047+0000","last_active":"2023-07-04T18:52:21.470569+0000","last_peered":"2023-07-04T18:52:21.470569+0000","last_clean":"2023-07-04T18:52:21.470569+0000","last_became_active":"2023-07-04T18:52:19.465676+0000","last_became_peered":"2023-07-04T18:52:19.465676+0000","last_unstale":"2023-07-04T18:52:21.470569+0000","last_undegraded":"2023-07-04T18:52:21.470569+0000","last_fullsized":"2023-07-04T18:52:21.470569+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:28:09.322921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470149+0000","last_change":"2023-07-04T18:52:20.486940+0000","last_active":"2023-07-04T18:52:21.470149+0000","last_peered":"2023-07-04T18:52:21.470149+0000","last_clean":"2023-07-04T18:52:21.470149+0000","last_became_active":"2023-07-04T18:52:20.486728+0000","last_became_peered":"2023-07-04T18:52:20.486728+0000","last_unstale":"2023-07-04T18:52:21.470149+0000","last_undegraded":"2023-07-04T18:52:21.470149+0000","last_fullsized":"2023-07-04T18:52:21.470149+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T20:38:33.470096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063999+0000","last_change":"2023-07-04T18:52:19.458036+0000","last_active":"2023-07-04T18:52:22.063999+0000","last_peered":"2023-07-04T18:52:22.063999+0000","last_clean":"2023-07-04T18:52:22.063999+0000","last_became_active":"2023-07-04T18:52:19.457785+0000","last_became_peered":"2023-07-04T18:52:19.457785+0000","last_unstale":"2023-07-04T18:52:22.063999+0000","last_undegraded":"2023-07-04T18:52:22.063999+0000","last_fullsized":"2023-07-04T18:52:22.063999+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:30:57.141214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064474+0000","last_change":"2023-07-04T18:52:20.479886+0000","last_active":"2023-07-04T18:52:22.064474+0000","last_peered":"2023-07-04T18:52:22.064474+0000","last_clean":"2023-07-04T18:52:22.064474+0000","last_became_active":"2023-07-04T18:52:20.479590+0000","last_became_peered":"2023-07-04T18:52:20.479590+0000","last_unstale":"2023-07-04T18:52:22.064474+0000","last_undegraded":"2023-07-04T18:52:22.064474+0000","last_fullsized":"2023-07-04T18:52:22.064474+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:15.554299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469340+0000","last_change":"2023-07-04T18:52:19.456613+0000","last_active":"2023-07-04T18:52:21.469340+0000","last_peered":"2023-07-04T18:52:21.469340+0000","last_clean":"2023-07-04T18:52:21.469340+0000","last_became_active":"2023-07-04T18:52:19.456323+0000","last_became_peered":"2023-07-04T18:52:19.456323+0000","last_unstale":"2023-07-04T18:52:21.469340+0000","last_undegraded":"2023-07-04T18:52:21.469340+0000","last_fullsized":"2023-07-04T18:52:21.469340+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:09:03.132735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470266+0000","last_change":"2023-07-04T18:52:20.486461+0000","last_active":"2023-07-04T18:52:21.470266+0000","last_peered":"2023-07-04T18:52:21.470266+0000","last_clean":"2023-07-04T18:52:21.470266+0000","last_became_active":"2023-07-04T18:52:20.486131+0000","last_became_peered":"2023-07-04T18:52:20.486131+0000","last_unstale":"2023-07-04T18:52:21.470266+0000","last_undegraded":"2023-07-04T18:52:21.470266+0000","last_fullsized":"2023-07-04T18:52:21.470266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:01.357747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"36'8","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064559+0000","last_change":"2023-07-04T18:52:19.464608+0000","last_active":"2023-07-04T18:52:22.064559+0000","last_peered":"2023-07-04T18:52:22.064559+0000","last_clean":"2023-07-04T18:52:22.064559+0000","last_became_active":"2023-07-04T18:52:19.464317+0000","last_became_peered":"2023-07-04T18:52:19.464317+0000","last_unstale":"2023-07-04T18:52:22.064559+0000","last_undegraded":"2023-07-04T18:52:22.064559+0000","last_fullsized":"2023-07-04T18:52:22.064559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:57:52.218872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470488+0000","last_change":"2023-07-04T18:52:20.484585+0000","last_active":"2023-07-04T18:52:21.470488+0000","last_peered":"2023-07-04T18:52:21.470488+0000","last_clean":"2023-07-04T18:52:21.470488+0000","last_became_active":"2023-07-04T18:52:20.484314+0000","last_became_peered":"2023-07-04T18:52:20.484314+0000","last_unstale":"2023-07-04T18:52:21.470488+0000","last_undegraded":"2023-07-04T18:52:21.470488+0000","last_fullsized":"2023-07-04T18:52:21.470488+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:06:27.571180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583814+0000","last_change":"2023-07-04T18:52:19.458147+0000","last_active":"2023-07-04T18:52:21.583814+0000","last_peered":"2023-07-04T18:52:21.583814+0000","last_clean":"2023-07-04T18:52:21.583814+0000","last_became_active":"2023-07-04T18:52:19.457690+0000","last_became_peered":"2023-07-04T18:52:19.457690+0000","last_unstale":"2023-07-04T18:52:21.583814+0000","last_undegraded":"2023-07-04T18:52:21.583814+0000","last_fullsized":"2023-07-04T18:52:21.583814+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:15:10.064186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583832+0000","last_change":"2023-07-04T18:52:20.492732+0000","last_active":"2023-07-04T18:52:21.583832+0000","last_peered":"2023-07-04T18:52:21.583832+0000","last_clean":"2023-07-04T18:52:21.583832+0000","last_became_active":"2023-07-04T18:52:20.490197+0000","last_became_peered":"2023-07-04T18:52:20.490197+0000","last_unstale":"2023-07-04T18:52:21.583832+0000","last_undegraded":"2023-07-04T18:52:21.583832+0000","last_fullsized":"2023-07-04T18:52:21.583832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:16:43.486666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471601+0000","last_change":"2023-07-04T18:52:19.455963+0000","last_active":"2023-07-04T18:52:21.471601+0000","last_peered":"2023-07-04T18:52:21.471601+0000","last_clean":"2023-07-04T18:52:21.471601+0000","last_became_active":"2023-07-04T18:52:19.455693+0000","last_became_peered":"2023-07-04T18:52:19.455693+0000","last_unstale":"2023-07-04T18:52:21.471601+0000","last_undegraded":"2023-07-04T18:52:21.471601+0000","last_fullsized":"2023-07-04T18:52:21.471601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T04:14:46.933898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471710+0000","last_change":"2023-07-04T18:52:20.479233+0000","last_active":"2023-07-04T18:52:21.471710+0000","last_peered":"2023-07-04T18:52:21.471710+0000","last_clean":"2023-07-04T18:52:21.471710+0000","last_became_active":"2023-07-04T18:52:20.478983+0000","last_became_peered":"2023-07-04T18:52:20.478983+0000","last_unstale":"2023-07-04T18:52:21.471710+0000","last_undegraded":"2023-07-04T18:52:21.471710+0000","last_fullsized":"2023-07-04T18:52:21.471710+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:28:52.986168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"36'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581163+0000","last_change":"2023-07-04T18:52:19.462190+0000","last_active":"2023-07-04T18:52:21.581163+0000","last_peered":"2023-07-04T18:52:21.581163+0000","last_clean":"2023-07-04T18:52:21.581163+0000","last_became_active":"2023-07-04T18:52:19.461906+0000","last_became_peered":"2023-07-04T18:52:19.461906+0000","last_unstale":"2023-07-04T18:52:21.581163+0000","last_undegraded":"2023-07-04T18:52:21.581163+0000","last_fullsized":"2023-07-04T18:52:21.581163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:48.049300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581307+0000","last_change":"2023-07-04T18:52:20.479758+0000","last_active":"2023-07-04T18:52:21.581307+0000","last_peered":"2023-07-04T18:52:21.581307+0000","last_clean":"2023-07-04T18:52:21.581307+0000","last_became_active":"2023-07-04T18:52:20.479319+0000","last_became_peered":"2023-07-04T18:52:20.479319+0000","last_unstale":"2023-07-04T18:52:21.581307+0000","last_undegraded":"2023-07-04T18:52:21.581307+0000","last_fullsized":"2023-07-04T18:52:21.581307+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:28:00.720148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063995+0000","last_change":"2023-07-04T18:52:19.457634+0000","last_active":"2023-07-04T18:52:22.063995+0000","last_peered":"2023-07-04T18:52:22.063995+0000","last_clean":"2023-07-04T18:52:22.063995+0000","last_became_active":"2023-07-04T18:52:19.457341+0000","last_became_peered":"2023-07-04T18:52:19.457341+0000","last_unstale":"2023-07-04T18:52:22.063995+0000","last_undegraded":"2023-07-04T18:52:22.063995+0000","last_fullsized":"2023-07-04T18:52:22.063995+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:35:03.681656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064181+0000","last_change":"2023-07-04T18:52:20.483273+0000","last_active":"2023-07-04T18:52:22.064181+0000","last_peered":"2023-07-04T18:52:22.064181+0000","last_clean":"2023-07-04T18:52:22.064181+0000","last_became_active":"2023-07-04T18:52:20.480069+0000","last_became_peered":"2023-07-04T18:52:20.480069+0000","last_unstale":"2023-07-04T18:52:22.064181+0000","last_undegraded":"2023-07-04T18:52:22.064181+0000","last_fullsized":"2023-07-04T18:52:22.064181+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581171+0000","last_change":"2023-07-04T18:52:19.458392+0000","last_active":"2023-07-04T18:52:21.581171+0000","last_peered":"2023-07-04T18:52:21.581171+0000","last_clean":"2023-07-04T18:52:21.581171+0000","last_became_active":"2023-07-04T18:52:19.458046+0000","last_became_peered":"2023-07-04T18:52:19.458046+0000","last_unstale":"2023-07-04T18:52:21.581171+0000","last_undegraded":"2023-07-04T18:52:21.581171+0000","last_fullsized":"2023-07-04T18:52:21.581171+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:40:12.555311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063694+0000","last_change":"2023-07-04T18:52:20.482831+0000","last_active":"2023-07-04T18:52:22.063694+0000","last_peered":"2023-07-04T18:52:22.063694+0000","last_clean":"2023-07-04T18:52:22.063694+0000","last_became_active":"2023-07-04T18:52:20.482494+0000","last_became_peered":"2023-07-04T18:52:20.482494+0000","last_unstale":"2023-07-04T18:52:22.063694+0000","last_undegraded":"2023-07-04T18:52:22.063694+0000","last_fullsized":"2023-07-04T18:52:22.063694+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:45:42.643724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581048+0000","last_change":"2023-07-04T18:52:19.456664+0000","last_active":"2023-07-04T18:52:21.581048+0000","last_peered":"2023-07-04T18:52:21.581048+0000","last_clean":"2023-07-04T18:52:21.581048+0000","last_became_active":"2023-07-04T18:52:19.456374+0000","last_became_peered":"2023-07-04T18:52:19.456374+0000","last_unstale":"2023-07-04T18:52:21.581048+0000","last_undegraded":"2023-07-04T18:52:21.581048+0000","last_fullsized":"2023-07-04T18:52:21.581048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T05:04:05.770013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583383+0000","last_change":"2023-07-04T18:52:20.496556+0000","last_active":"2023-07-04T18:52:21.583383+0000","last_peered":"2023-07-04T18:52:21.583383+0000","last_clean":"2023-07-04T18:52:21.583383+0000","last_became_active":"2023-07-04T18:52:20.495594+0000","last_became_peered":"2023-07-04T18:52:20.495594+0000","last_unstale":"2023-07-04T18:52:21.583383+0000","last_undegraded":"2023-07-04T18:52:21.583383+0000","last_fullsized":"2023-07-04T18:52:21.583383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:29:33.232347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064681+0000","last_change":"2023-07-04T18:52:19.466675+0000","last_active":"2023-07-04T18:52:22.064681+0000","last_peered":"2023-07-04T18:52:22.064681+0000","last_clean":"2023-07-04T18:52:22.064681+0000","last_became_active":"2023-07-04T18:52:19.466465+0000","last_became_peered":"2023-07-04T18:52:19.466465+0000","last_unstale":"2023-07-04T18:52:22.064681+0000","last_undegraded":"2023-07-04T18:52:22.064681+0000","last_fullsized":"2023-07-04T18:52:22.064681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:38:13.573589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470404+0000","last_change":"2023-07-04T18:52:20.481606+0000","last_active":"2023-07-04T18:52:21.470404+0000","last_peered":"2023-07-04T18:52:21.470404+0000","last_clean":"2023-07-04T18:52:21.470404+0000","last_became_active":"2023-07-04T18:52:20.481341+0000","last_became_peered":"2023-07-04T18:52:20.481341+0000","last_unstale":"2023-07-04T18:52:21.470404+0000","last_undegraded":"2023-07-04T18:52:21.470404+0000","last_fullsized":"2023-07-04T18:52:21.470404+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:58.532607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581050+0000","last_change":"2023-07-04T18:52:19.457999+0000","last_active":"2023-07-04T18:52:21.581050+0000","last_peered":"2023-07-04T18:52:21.581050+0000","last_clean":"2023-07-04T18:52:21.581050+0000","last_became_active":"2023-07-04T18:52:19.457744+0000","last_became_peered":"2023-07-04T18:52:19.457744+0000","last_unstale":"2023-07-04T18:52:21.581050+0000","last_undegraded":"2023-07-04T18:52:21.581050+0000","last_fullsized":"2023-07-04T18:52:21.581050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:08:10.842309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063778+0000","last_change":"2023-07-04T18:52:20.485157+0000","last_active":"2023-07-04T18:52:22.063778+0000","last_peered":"2023-07-04T18:52:22.063778+0000","last_clean":"2023-07-04T18:52:22.063778+0000","last_became_active":"2023-07-04T18:52:20.484906+0000","last_became_peered":"2023-07-04T18:52:20.484906+0000","last_unstale":"2023-07-04T18:52:22.063778+0000","last_undegraded":"2023-07-04T18:52:22.063778+0000","last_fullsized":"2023-07-04T18:52:22.063778+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:02:28.309699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064086+0000","last_change":"2023-07-04T18:52:19.461058+0000","last_active":"2023-07-04T18:52:22.064086+0000","last_peered":"2023-07-04T18:52:22.064086+0000","last_clean":"2023-07-04T18:52:22.064086+0000","last_became_active":"2023-07-04T18:52:19.460786+0000","last_became_peered":"2023-07-04T18:52:19.460786+0000","last_unstale":"2023-07-04T18:52:22.064086+0000","last_undegraded":"2023-07-04T18:52:22.064086+0000","last_fullsized":"2023-07-04T18:52:22.064086+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:43:57.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064059+0000","last_change":"2023-07-04T18:51:18.208348+0000","last_active":"2023-07-04T18:52:22.064059+0000","last_peered":"2023-07-04T18:52:22.064059+0000","last_clean":"2023-07-04T18:52:22.064059+0000","last_became_active":"2023-07-04T18:51:18.208032+0000","last_became_peered":"2023-07-04T18:51:18.208032+0000","last_unstale":"2023-07-04T18:52:22.064059+0000","last_undegraded":"2023-07-04T18:52:22.064059+0000","last_fullsized":"2023-07-04T18:52:22.064059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:45:06.368879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470452+0000","last_change":"2023-07-04T18:52:20.491037+0000","last_active":"2023-07-04T18:52:21.470452+0000","last_peered":"2023-07-04T18:52:21.470452+0000","last_clean":"2023-07-04T18:52:21.470452+0000","last_became_active":"2023-07-04T18:52:20.490723+0000","last_became_peered":"2023-07-04T18:52:20.490723+0000","last_unstale":"2023-07-04T18:52:21.470452+0000","last_undegraded":"2023-07-04T18:52:21.470452+0000","last_fullsized":"2023-07-04T18:52:21.470452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T03:57:58.886948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063944+0000","last_change":"2023-07-04T18:52:19.462323+0000","last_active":"2023-07-04T18:52:22.063944+0000","last_peered":"2023-07-04T18:52:22.063944+0000","last_clean":"2023-07-04T18:52:22.063944+0000","last_became_active":"2023-07-04T18:52:19.462057+0000","last_became_peered":"2023-07-04T18:52:19.462057+0000","last_unstale":"2023-07-04T18:52:22.063944+0000","last_undegraded":"2023-07-04T18:52:22.063944+0000","last_fullsized":"2023-07-04T18:52:22.063944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:06:00.356116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583488+0000","last_change":"2023-07-04T18:51:17.318399+0000","last_active":"2023-07-04T18:52:21.583488+0000","last_peered":"2023-07-04T18:52:21.583488+0000","last_clean":"2023-07-04T18:52:21.583488+0000","last_became_active":"2023-07-04T18:51:17.317923+0000","last_became_peered":"2023-07-04T18:51:17.317923+0000","last_unstale":"2023-07-04T18:52:21.583488+0000","last_undegraded":"2023-07-04T18:52:21.583488+0000","last_fullsized":"2023-07-04T18:52:21.583488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:21:22.798814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064397+0000","last_change":"2023-07-04T18:52:20.484156+0000","last_active":"2023-07-04T18:52:22.064397+0000","last_peered":"2023-07-04T18:52:22.064397+0000","last_clean":"2023-07-04T18:52:22.064397+0000","last_became_active":"2023-07-04T18:52:20.483878+0000","last_became_peered":"2023-07-04T18:52:20.483878+0000","last_unstale":"2023-07-04T18:52:22.064397+0000","last_undegraded":"2023-07-04T18:52:22.064397+0000","last_fullsized":"2023-07-04T18:52:22.064397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T23:06:27.030720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"37'1","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581165+0000","last_change":"2023-07-04T18:52:19.460102+0000","last_active":"2023-07-04T18:52:21.581165+0000","last_peered":"2023-07-04T18:52:21.581165+0000","last_clean":"2023-07-04T18:52:21.581165+0000","last_became_active":"2023-07-04T18:52:19.459830+0000","last_became_peered":"2023-07-04T18:52:19.459830+0000","last_unstale":"2023-07-04T18:52:21.581165+0000","last_undegraded":"2023-07-04T18:52:21.581165+0000","last_fullsized":"2023-07-04T18:52:21.581165+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:17:54.321667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.581016+0000","last_change":"2023-07-04T18:51:17.315168+0000","last_active":"2023-07-04T18:52:21.581016+0000","last_peered":"2023-07-04T18:52:21.581016+0000","last_clean":"2023-07-04T18:52:21.581016+0000","last_became_active":"2023-07-04T18:51:17.314862+0000","last_became_peered":"2023-07-04T18:51:17.314862+0000","last_unstale":"2023-07-04T18:52:21.581016+0000","last_undegraded":"2023-07-04T18:52:21.581016+0000","last_fullsized":"2023-07-04T18:52:21.581016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:13:46.966327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583836+0000","last_change":"2023-07-04T18:52:20.489978+0000","last_active":"2023-07-04T18:52:21.583836+0000","last_peered":"2023-07-04T18:52:21.583836+0000","last_clean":"2023-07-04T18:52:21.583836+0000","last_became_active":"2023-07-04T18:52:20.489690+0000","last_became_peered":"2023-07-04T18:52:20.489690+0000","last_unstale":"2023-07-04T18:52:21.583836+0000","last_undegraded":"2023-07-04T18:52:21.583836+0000","last_fullsized":"2023-07-04T18:52:21.583836+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:49:57.480646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064479+0000","last_change":"2023-07-04T18:52:19.469850+0000","last_active":"2023-07-04T18:52:22.064479+0000","last_peered":"2023-07-04T18:52:22.064479+0000","last_clean":"2023-07-04T18:52:22.064479+0000","last_became_active":"2023-07-04T18:52:19.469546+0000","last_became_peered":"2023-07-04T18:52:19.469546+0000","last_unstale":"2023-07-04T18:52:22.064479+0000","last_undegraded":"2023-07-04T18:52:22.064479+0000","last_fullsized":"2023-07-04T18:52:22.064479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:28:10.562186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583631+0000","last_change":"2023-07-04T18:51:17.318473+0000","last_active":"2023-07-04T18:52:21.583631+0000","last_peered":"2023-07-04T18:52:21.583631+0000","last_clean":"2023-07-04T18:52:21.583631+0000","last_became_active":"2023-07-04T18:51:17.317839+0000","last_became_peered":"2023-07-04T18:51:17.317839+0000","last_unstale":"2023-07-04T18:52:21.583631+0000","last_undegraded":"2023-07-04T18:52:21.583631+0000","last_fullsized":"2023-07-04T18:52:21.583631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:49.853323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":197,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064304+0000","last_change":"2023-07-04T18:51:11.231295+0000","last_active":"2023-07-04T18:52:22.064304+0000","last_peered":"2023-07-04T18:52:22.064304+0000","last_clean":"2023-07-04T18:52:22.064304+0000","last_became_active":"2023-07-04T18:51:11.230103+0000","last_became_peered":"2023-07-04T18:51:11.230103+0000","last_unstale":"2023-07-04T18:52:22.064304+0000","last_undegraded":"2023-07-04T18:52:22.064304+0000","last_fullsized":"2023-07-04T18:52:22.064304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:10.136401+0000","last_clean_scrub_stamp":"2023-07-04T18:51:10.136401+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:27:43.685172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063939+0000","last_change":"2023-07-04T18:52:20.497660+0000","last_active":"2023-07-04T18:52:22.063939+0000","last_peered":"2023-07-04T18:52:22.063939+0000","last_clean":"2023-07-04T18:52:22.063939+0000","last_became_active":"2023-07-04T18:52:20.497337+0000","last_became_peered":"2023-07-04T18:52:20.497337+0000","last_unstale":"2023-07-04T18:52:22.063939+0000","last_undegraded":"2023-07-04T18:52:22.063939+0000","last_fullsized":"2023-07-04T18:52:22.063939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:20:18.389953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"36'2","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.583996+0000","last_change":"2023-07-04T18:52:19.461997+0000","last_active":"2023-07-04T18:52:21.583996+0000","last_peered":"2023-07-04T18:52:21.583996+0000","last_clean":"2023-07-04T18:52:21.583996+0000","last_became_active":"2023-07-04T18:52:19.461815+0000","last_became_peered":"2023-07-04T18:52:19.461815+0000","last_unstale":"2023-07-04T18:52:21.583996+0000","last_undegraded":"2023-07-04T18:52:21.583996+0000","last_fullsized":"2023-07-04T18:52:21.583996+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:48:14.552668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064565+0000","last_change":"2023-07-04T18:51:17.318248+0000","last_active":"2023-07-04T18:52:22.064565+0000","last_peered":"2023-07-04T18:52:22.064565+0000","last_clean":"2023-07-04T18:52:22.064565+0000","last_became_active":"2023-07-04T18:51:17.317921+0000","last_became_peered":"2023-07-04T18:51:17.317921+0000","last_unstale":"2023-07-04T18:52:22.064565+0000","last_undegraded":"2023-07-04T18:52:22.064565+0000","last_fullsized":"2023-07-04T18:52:22.064565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T06:01:20.555239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.063962+0000","last_change":"2023-07-04T18:52:20.477535+0000","last_active":"2023-07-04T18:52:22.063962+0000","last_peered":"2023-07-04T18:52:22.063962+0000","last_clean":"2023-07-04T18:52:22.063962+0000","last_became_active":"2023-07-04T18:52:20.477247+0000","last_became_peered":"2023-07-04T18:52:20.477247+0000","last_unstale":"2023-07-04T18:52:22.063962+0000","last_undegraded":"2023-07-04T18:52:22.063962+0000","last_fullsized":"2023-07-04T18:52:22.063962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T19:00:07.153970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.469531+0000","last_change":"2023-07-04T18:52:19.455490+0000","last_active":"2023-07-04T18:52:21.469531+0000","last_peered":"2023-07-04T18:52:21.469531+0000","last_clean":"2023-07-04T18:52:21.469531+0000","last_became_active":"2023-07-04T18:52:19.455224+0000","last_became_peered":"2023-07-04T18:52:19.455224+0000","last_unstale":"2023-07-04T18:52:21.469531+0000","last_undegraded":"2023-07-04T18:52:21.469531+0000","last_fullsized":"2023-07-04T18:52:21.469531+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:53:35.991529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064762+0000","last_change":"2023-07-04T18:51:18.208558+0000","last_active":"2023-07-04T18:52:22.064762+0000","last_peered":"2023-07-04T18:52:22.064762+0000","last_clean":"2023-07-04T18:52:22.064762+0000","last_became_active":"2023-07-04T18:51:18.208239+0000","last_became_peered":"2023-07-04T18:51:18.208239+0000","last_unstale":"2023-07-04T18:52:22.064762+0000","last_undegraded":"2023-07-04T18:52:22.064762+0000","last_fullsized":"2023-07-04T18:52:22.064762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T01:26:23.252619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471172+0000","last_change":"2023-07-04T18:52:20.487224+0000","last_active":"2023-07-04T18:52:21.471172+0000","last_peered":"2023-07-04T18:52:21.471172+0000","last_clean":"2023-07-04T18:52:21.471172+0000","last_became_active":"2023-07-04T18:52:20.486638+0000","last_became_peered":"2023-07-04T18:52:20.486638+0000","last_unstale":"2023-07-04T18:52:21.471172+0000","last_undegraded":"2023-07-04T18:52:21.471172+0000","last_fullsized":"2023-07-04T18:52:21.471172+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:05:02.193514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470860+0000","last_change":"2023-07-04T18:52:19.465554+0000","last_active":"2023-07-04T18:52:21.470860+0000","last_peered":"2023-07-04T18:52:21.470860+0000","last_clean":"2023-07-04T18:52:21.470860+0000","last_became_active":"2023-07-04T18:52:19.465268+0000","last_became_peered":"2023-07-04T18:52:19.465268+0000","last_unstale":"2023-07-04T18:52:21.470860+0000","last_undegraded":"2023-07-04T18:52:21.470860+0000","last_fullsized":"2023-07-04T18:52:21.470860+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:22:24.930028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064543+0000","last_change":"2023-07-04T18:51:17.313573+0000","last_active":"2023-07-04T18:52:22.064543+0000","last_peered":"2023-07-04T18:52:22.064543+0000","last_clean":"2023-07-04T18:52:22.064543+0000","last_became_active":"2023-07-04T18:51:17.312995+0000","last_became_peered":"2023-07-04T18:51:17.312995+0000","last_unstale":"2023-07-04T18:52:22.064543+0000","last_undegraded":"2023-07-04T18:52:22.064543+0000","last_fullsized":"2023-07-04T18:52:22.064543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T22:28:30.099502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:22.064698+0000","last_change":"2023-07-04T18:52:20.479835+0000","last_active":"2023-07-04T18:52:22.064698+0000","last_peered":"2023-07-04T18:52:22.064698+0000","last_clean":"2023-07-04T18:52:22.064698+0000","last_became_active":"2023-07-04T18:52:20.479561+0000","last_became_peered":"2023-07-04T18:52:20.479561+0000","last_unstale":"2023-07-04T18:52:22.064698+0000","last_undegraded":"2023-07-04T18:52:22.064698+0000","last_fullsized":"2023-07-04T18:52:22.064698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:19.437396+0000","last_clean_scrub_stamp":"2023-07-04T18:52:19.437396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-05T21:59:27.432907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.470121+0000","last_change":"2023-07-04T18:52:19.461537+0000","last_active":"2023-07-04T18:52:21.470121+0000","last_peered":"2023-07-04T18:52:21.470121+0000","last_clean":"2023-07-04T18:52:21.470121+0000","last_became_active":"2023-07-04T18:52:19.461279+0000","last_became_peered":"2023-07-04T18:52:19.461279+0000","last_unstale":"2023-07-04T18:52:21.470121+0000","last_undegraded":"2023-07-04T18:52:21.470121+0000","last_fullsized":"2023-07-04T18:52:21.470121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:52:18.435765+0000","last_clean_scrub_stamp":"2023-07-04T18:52:18.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T00:17:08.890649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-07-04T18:52:21.471908+0000","last_change":"2023-07-04T18:51:17.315629+0000","last_active":"2023-07-04T18:52:21.471908+0000","last_peered":"2023-07-04T18:52:21.471908+0000","last_clean":"2023-07-04T18:52:21.471908+0000","last_became_active":"2023-07-04T18:51:17.315288+0000","last_became_peered":"2023-07-04T18:51:17.315288+0000","last_unstale":"2023-07-04T18:52:21.471908+0000","last_undegraded":"2023-07-04T18:52:21.471908+0000","last_fullsized":"2023-07-04T18:52:21.471908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-04T18:51:16.304551+0000","last_clean_scrub_stamp":"2023-07-04T18:51:16.304551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-06T02:11:33.840191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":1032192,"data_stored":767390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.496}]},{"osd":1,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":3.1379999999999999,"5min":3.1379999999999999,"15min":3.1379999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.59399999999999997}]},{"osd":3,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":3.262,"5min":3.262,"15min":3.262},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.66100000000000003}]},{"osd":4,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.69799999999999995}]},{"osd":5,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68400000000000005}]},{"osd":6,"last update":"Tue Jul 4 18:52:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.74399999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.63200000000000001}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.73099999999999998}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.61699999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.499},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46500000000000002}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.505},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42199999999999999}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.988,"5min":2.988,"15min":2.988},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.0600000000000001,"5min":3.0600000000000001,"15min":3.0600000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.73099999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.70299999999999996}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.4889999999999999,"5min":3.4889999999999999,"15min":3.4889999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.76400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":1019904,"data_stored":765790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.218},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.254},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.6720000000000002},"last":0.29499999999999998}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.35699999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.7719999999999998,"5min":3.7719999999999998,"15min":3.7719999999999998},"last":0.622},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56599999999999995}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":3.7109999999999999,"5min":3.7109999999999999,"15min":3.7109999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.57499999999999996}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.742,"5min":3.742,"15min":3.742},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":425984,"data_stored":175458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.42199999999999999}]},{"osd":1,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.307},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.34100000000000003}]},{"osd":3,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.378}]},{"osd":4,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.439}]},{"osd":6,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48999999999999999}]},{"osd":7,"last update":"Tue Jul 4 18:52:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.502},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.47799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":438272,"data_stored":176986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.315}]},{"osd":1,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27600000000000002}]},{"osd":2,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.54200000000000004}]},{"osd":5,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.65600000000000003}]},{"osd":6,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71099999999999997}]},{"osd":7,"last update":"Tue Jul 4 18:52:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.47599999999999998}]},{"osd":5,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40400000000000003}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.34000000000000002}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.439}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":430080,"data_stored":175500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.629},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70199999999999996}]},{"osd":3,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.72899999999999998}]},{"osd":4,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.38},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.41499999999999998}]},{"osd":6,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.438}]},{"osd":7,"last update":"Tue Jul 4 18:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-04T18:52:29.497 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-04T18:52:29.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-07-04T18:52:29.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.650+0000 7fc8792db700 1 Processor -- start 2023-07-04T18:52:29.652 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.650+0000 7fc8792db700 1 -- start start 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8792db700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc874158f80 0x7fc87406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8792db700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87406ccb0 con 0x7fc874159350 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8792db700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87406cf90 con 0x7fc8740cd590 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8792db700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87406d270 con 0x7fc874158f80 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc872ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc874158f80 0x7fc87406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8727fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc874159350 0x7fc87406c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53202/0 (socket says 172.21.15.77:53202) 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.651+0000 7fc8727fc700 1 -- 172.21.15.77:0/1949331979 learned_addr learned my addr 172.21.15.77:0/1949331979 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc872ffd700 1 -- 172.21.15.77:0/1949331979 >> v1:172.21.15.82:6789/0 conn(0x7fc8740cd590 legacy=0x7fc8740ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc872ffd700 1 -- 172.21.15.77:0/1949331979 >> v1:172.21.15.77:6789/0 conn(0x7fc874159350 legacy=0x7fc87406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc872ffd700 1 -- 172.21.15.77:0/1949331979 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87406d550 con 0x7fc874158f80 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc871ffb700 1 -- 172.21.15.77:0/1949331979 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874448678 0 0) 0x7fc87406ccb0 con 0x7fc874159350 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc872ffd700 1 --2- 172.21.15.77:0/1949331979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874158f80 0x7fc87406dd50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc868009840 tx=0x7fc868009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5697dfbf3f670b47 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc871ffb700 1 -- 172.21.15.77:0/1949331979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868004230 con 0x7fc874158f80 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc871ffb700 1 -- 172.21.15.77:0/1949331979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc868004390 con 0x7fc874158f80 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc871ffb700 1 -- 172.21.15.77:0/1949331979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8680036a0 con 0x7fc874158f80 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc8792db700 1 -- 172.21.15.77:0/1949331979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874158f80 msgr2=0x7fc87406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc8792db700 1 --2- 172.21.15.77:0/1949331979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874158f80 0x7fc87406dd50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc868009840 tx=0x7fc868009b50 comp rx=0 tx=0).stop 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc8792db700 1 -- 172.21.15.77:0/1949331979 shutdown_connections 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc8792db700 1 --2- 172.21.15.77:0/1949331979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc874158f80 0x7fc87406dd50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.652+0000 7fc8792db700 1 -- 172.21.15.77:0/1949331979 >> 172.21.15.77:0/1949331979 conn(0x7fc87406bba0 msgr2=0x7fc8740b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.653+0000 7fc8792db700 1 -- 172.21.15.77:0/1949331979 shutdown_connections 2023-07-04T18:52:29.657 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.653+0000 7fc8792db700 1 -- 172.21.15.77:0/1949331979 wait complete. 2023-07-04T18:52:29.658 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.654+0000 7fc8792db700 1 Processor -- start 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.654+0000 7fc8792db700 1 -- start start 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 0x7fc874069e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 0x7fc87406a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740c2cf0 con 0x7fc8740cd590 2023-07-04T18:52:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874153720 con 0x7fc874158f80 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc8792db700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc874153a00 con 0x7fc874159350 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc872ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46068/0 (socket says 172.21.15.77:46068) 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.655+0000 7fc872ffd700 1 -- 172.21.15.77:0/1601417644 learned_addr learned my addr 172.21.15.77:0/1601417644 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8737fe700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 0x7fc87406a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 -- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 msgr2=0x7fc87406a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 0x7fc87406a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 -- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 msgr2=0x7fc874069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8727fc700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 0x7fc874069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 0x7fc874069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 -- 172.21.15.77:0/1601417644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8680094a0 con 0x7fc8740cd590 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8737fe700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 0x7fc87406a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8727fc700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 0x7fc874069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc872ffd700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc8740d0ac0 tx=0x7fc868004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc868011630 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86801e070 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8680119f0 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc874153ce0 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.656+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8741542a0 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.657+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc874154490 con 0x7fc8740cd590 2023-07-04T18:52:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.657+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc868011b50 con 0x7fc8740cd590 2023-07-04T18:52:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.658+0000 7fc85b7fe700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 0x7fc86006d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.658+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fc86808a830 con 0x7fc8740cd590 2023-07-04T18:52:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.658+0000 7fc8727fc700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 0x7fc86006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.658+0000 7fc8727fc700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 0x7fc86006d160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc87413dac0 tx=0x7fc864009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.662+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc868055970 con 0x7fc8740cd590 2023-07-04T18:52:29.927 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.925+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc87404e210 con 0x7fc8740cd590 2023-07-04T18:52:29.934 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.932+0000 7fc85b7fe700 1 -- 172.21.15.77:0/1601417644 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc8680200e0 con 0x7fc8740cd590 2023-07-04T18:52:29.935 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2023-07-04T18:52:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDwaaRkpsg9CxAA3I+2BiR3flfH2NGjwz9V9Q== 2023-07-04T18:52:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: caps mds = "allow" 2023-07-04T18:52:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "allow r" 2023-07-04T18:52:29.935 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-07-04T18:52:29.936 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 msgr2=0x7fc86006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 0x7fc86006d160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc87413dac0 tx=0x7fc864009040 comp rx=0 tx=0).stop 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 msgr2=0x7fc8740c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc8740d0ac0 tx=0x7fc868004750 comp rx=0 tx=0).stop 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 shutdown_connections 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fc86006acb0 0x7fc86006d160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8740cd590 0x7fc8740c27b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.937 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fc874159350 0x7fc87406a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 --2- 172.21.15.77:0/1601417644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc874158f80 0x7fc874069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:29.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 >> 172.21.15.77:0/1601417644 conn(0x7fc87406bba0 msgr2=0x7fc874158190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:29.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 shutdown_connections 2023-07-04T18:52:29.938 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:29.934+0000 7fc8792db700 1 -- 172.21.15.77:0/1601417644 wait complete. 2023-07-04T18:52:29.938 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.0 2023-07-04T18:52:29.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-04T18:52:30.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.067+0000 7fe9ab2a6700 1 Processor -- start 2023-07-04T18:52:30.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 -- start start 2023-07-04T18:52:30.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 0x7fe9a40cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.073 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9a415ca10 0x7fe9a4070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a4070db0 0x7fe9a40cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a4071210 con 0x7fe9a415ca10 2023-07-04T18:52:30.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a40717f0 con 0x7fe9a415c640 2023-07-04T18:52:30.074 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.068+0000 7fe9ab2a6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40cd290 con 0x7fe9a4070db0 2023-07-04T18:52:30.079 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9a415ca10 0x7fe9a4070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.079 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe9a415ca10 0x7fe9a4070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46076/0 (socket says 172.21.15.77:46076) 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 -- 172.21.15.77:0/1392327724 learned_addr learned my addr 172.21.15.77:0/1392327724 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a9843700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a4070db0 0x7fe9a40cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a9042700 1 --2- 172.21.15.77:0/1392327724 >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 0x7fe9a40cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 -- 172.21.15.77:0/1392327724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a4070db0 msgr2=0x7fe9a40cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a4070db0 0x7fe9a40cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.080 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 -- 172.21.15.77:0/1392327724 >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 msgr2=0x7fe9a40cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 --2- 172.21.15.77:0/1392327724 >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 0x7fe9a40cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 -- 172.21.15.77:0/1392327724 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40ccfb0 con 0x7fe9a415ca10 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a9042700 1 --2- 172.21.15.77:0/1392327724 >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 0x7fe9a40cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe9a8841700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a415ca10 0x7fe9a4070870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4069540 tx=0x7fe9a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14b230932a87ea6 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe99bfff700 1 -- 172.21.15.77:0/1392327724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0011830 con 0x7fe9a415ca10 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe99bfff700 1 -- 172.21.15.77:0/1392327724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a0011990 con 0x7fe9a415ca10 2023-07-04T18:52:30.081 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.069+0000 7fe99bfff700 1 -- 172.21.15.77:0/1392327724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0011d20 con 0x7fe9a415ca10 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/1392327724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a415ca10 msgr2=0x7fe9a4070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a415ca10 0x7fe9a4070870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4069540 tx=0x7fe9a0009bd0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/1392327724 shutdown_connections 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a4070db0 0x7fe9a40cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/1392327724 >> v2:172.21.15.82:3300/0 conn(0x7fe9a415c640 0x7fe9a40cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/1392327724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a415ca10 0x7fe9a4070870 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/1392327724 >> 172.21.15.77:0/1392327724 conn(0x7fe9a406bba0 msgr2=0x7fe9a40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.082 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/1392327724 shutdown_connections 2023-07-04T18:52:30.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.070+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/1392327724 wait complete. 2023-07-04T18:52:30.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.071+0000 7fe9ab2a6700 1 Processor -- start 2023-07-04T18:52:30.083 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.071+0000 7fe9ab2a6700 1 -- start start 2023-07-04T18:52:30.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.085 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 0x7fe9a40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a415ca10 0x7fe9a40cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40c8cf0 con 0x7fe9a4070db0 2023-07-04T18:52:30.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40b4b80 con 0x7fe9a415c640 2023-07-04T18:52:30.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9ab2a6700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40b4e60 con 0x7fe9a415ca10 2023-07-04T18:52:30.086 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46080/0 (socket says 172.21.15.77:46080) 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 -- 172.21.15.77:0/2587417244 learned_addr learned my addr 172.21.15.77:0/2587417244 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9843700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a415ca10 0x7fe9a40cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a8841700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 0x7fe9a40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 -- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a415ca10 msgr2=0x7fe9a40cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a415ca10 0x7fe9a40cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 -- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 msgr2=0x7fe9a40cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.087 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 0x7fe9a40cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 -- 172.21.15.77:0/2587417244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0009550 con 0x7fe9a4070db0 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a8841700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 0x7fe9a40cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.072+0000 7fe9a9042700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e760 tx=0x7fe99400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.073+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994013610 con 0x7fe9a4070db0 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.073+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994013770 con 0x7fe9a4070db0 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.073+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a40b5140 con 0x7fe9a4070db0 2023-07-04T18:52:30.088 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.073+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994013b20 con 0x7fe9a4070db0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.073+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a40b5700 con 0x7fe9a4070db0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.074+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a415aec0 con 0x7fe9a4070db0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.074+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe994013c80 con 0x7fe9a4070db0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.074+0000 7fe999ffb700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 0x7fe98806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.075+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fe994088f60 con 0x7fe9a4070db0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.075+0000 7fe9a8841700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 0x7fe98806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.075+0000 7fe9a8841700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 0x7fe98806d190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009c10 tx=0x7fe9a0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.089 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.079+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe994057aa0 con 0x7fe9a4070db0 2023-07-04T18:52:30.310 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.308+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9a404e210 con 0x7fe9a4070db0 2023-07-04T18:52:30.311 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.309+0000 7fe999ffb700 1 -- 172.21.15.77:0/2587417244 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4668 (secure 0 0 0) 0x7fe994057630 con 0x7fe9a4070db0 2023-07-04T18:52:30.311 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:30.311 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4784,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6836/1695626511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1695626511},{"type":"v1","addr":"172.21.15.82:6836","nonce":1695626511}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4903,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/3849345135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":3849345135},{"type":"v1","addr":"172.21.15.77:6835","nonce":3849345135}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/3698064560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3698064560},{"type":"v1","addr":"172.21.15.77:6834","nonce":3698064560}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-04T18:52:19.841356+0000","modified":"2023-07-04T18:52:23.480957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5136},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5136":{"gid":5136,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.82:6837/744114579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6835","nonce":744114579},{"type":"v1","addr":"172.21.15.82:6837","nonce":744114579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-07-04T18:52:30.312 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 msgr2=0x7fe98806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 0x7fe98806d190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009c10 tx=0x7fe9a0008030 comp rx=0 tx=0).stop 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 msgr2=0x7fe9a40b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e760 tx=0x7fe99400ea70 comp rx=0 tx=0).stop 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 shutdown_connections 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7fe98806ace0 0x7fe98806d190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9a4070db0 0x7fe9a40b42a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.313 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7fe9a415ca10 0x7fe9a40cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 --2- 172.21.15.77:0/2587417244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9a415c640 0x7fe9a40cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 >> 172.21.15.77:0/2587417244 conn(0x7fe9a406bba0 msgr2=0x7fe9a40b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.314 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 shutdown_connections 2023-07-04T18:52:30.315 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.311+0000 7fe9ab2a6700 1 -- 172.21.15.77:0/2587417244 wait complete. 2023-07-04T18:52:30.315 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2023-07-04T18:52:30.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-04T18:52:19.841356+0000', 'modified': '2023-07-04T18:52:23.480957+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5136}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5136': {'gid': 5136, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.82:6837/744114579', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6835', 'nonce': 744114579}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 744114579}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-04T18:52:30.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-04T18:52:30.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-07-04T18:52:30.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.849+0000 7f45341fd700 1 Processor -- start 2023-07-04T18:52:30.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.849+0000 7f45341fd700 1 -- start start 2023-07-04T18:52:30.851 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f45341fd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f452c070870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f45341fd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0cf830 con 0x7f452c15c640 2023-07-04T18:52:30.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f45341fd700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0717f0 con 0x7f452c070db0 2023-07-04T18:52:30.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f45341fd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c071250 con 0x7f452c15ca10 2023-07-04T18:52:30.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f4531f99700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f452c15c640 0x7f452c0cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53226/0 (socket says 172.21.15.77:53226) 2023-07-04T18:52:30.852 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.850+0000 7f4531f99700 1 -- 172.21.15.77:0/108245766 learned_addr learned my addr 172.21.15.77:0/108245766 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35174507 0 0) 0x7f452c0cf830 con 0x7f452c15c640 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4531798700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f452c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45180035b0 con 0x7f452c15c640 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2119138409 0 0) 0x7f45180035b0 con 0x7f452c15c640 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 msgr2=0x7f452c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.855 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f452c070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 >> v1:172.21.15.82:6789/0 conn(0x7f452c070db0 legacy=0x7f452c0cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0cd290 con 0x7f452c15c640 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3329947791 0 0) 0x7f4528001670 con 0x7f452c15c640 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 >> v1:172.21.15.77:6789/0 conn(0x7f452c15c640 legacy=0x7f452c0cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f4518003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4518003fb0 0x7f4518004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.856 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4518004d90 0x7f4518005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451800bbf0 con 0x7f4518004d90 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451800bd60 con 0x7f4518003fb0 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451800bf20 con 0x7f452c15ca10 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4518004d90 0x7f4518005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4531798700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4518003fb0 0x7f4518004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f4531f99700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f4518003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 -- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 msgr2=0x7f4518003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.857 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c15ca10 0x7f4518003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 -- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4518003fb0 msgr2=0x7f4518004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4518003fb0 0x7f4518004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.851+0000 7f453279a700 1 -- 172.21.15.77:0/108245766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180035b0 con 0x7f4518004d90 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f453279a700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4518004d90 0x7f4518005200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f452400c160 tx=0x7f4518005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e664e864bc1a5903 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524001db0 con 0x7f4518004d90 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452400cca0 con 0x7f4518004d90 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f4530f97700 1 -- 172.21.15.77:0/108245766 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524003c40 con 0x7f4518004d90 2023-07-04T18:52:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 -- 172.21.15.77:0/108245766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4518004d90 msgr2=0x7f4518005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 --2- 172.21.15.77:0/108245766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4518004d90 0x7f4518005200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f452400c160 tx=0x7f4518005880 comp rx=0 tx=0).stop 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f453279a700 1 -- 172.21.15.77:0/108245766 reap_dead start 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 -- 172.21.15.77:0/108245766 shutdown_connections 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 -- 172.21.15.77:0/108245766 >> 172.21.15.77:0/108245766 conn(0x7f452c06bba0 msgr2=0x7f452c0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 -- 172.21.15.77:0/108245766 shutdown_connections 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.852+0000 7f45341fd700 1 -- 172.21.15.77:0/108245766 wait complete. 2023-07-04T18:52:30.859 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.853+0000 7f45341fd700 1 Processor -- start 2023-07-04T18:52:30.860 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 -- start start 2023-07-04T18:52:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452c0b0960 0x7f452c158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c159810 0x7f452c1f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c1593f0 con 0x7f452c159c80 2023-07-04T18:52:30.863 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0b3bc0 con 0x7f452c0b0960 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f45341fd700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c158d30 con 0x7f452c159810 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f453279a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f453279a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46104/0 (socket says 172.21.15.77:46104) 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.854+0000 7f453279a700 1 -- 172.21.15.77:0/2446140323 learned_addr learned my addr 172.21.15.77:0/2446140323 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 -- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c159810 msgr2=0x7f452c1f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c159810 0x7f452c1f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.864 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 -- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452c0b0960 msgr2=0x7f452c158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f4531f99700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452c0b0960 0x7f452c158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452c0b0960 0x7f452c158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 -- 172.21.15.77:0/2446140323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c159010 con 0x7f452c159c80 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f453279a700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4524015040 tx=0x7f45240014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f45227fc700 1 -- 172.21.15.77:0/2446140323 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524003e10 con 0x7f452c159c80 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f45227fc700 1 -- 172.21.15.77:0/2446140323 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45240050d0 con 0x7f452c159c80 2023-07-04T18:52:30.865 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c1f6490 con 0x7f452c159c80 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f45227fc700 1 -- 172.21.15.77:0/2446140323 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524005460 con 0x7f452c159c80 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.855+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c0b3850 con 0x7f452c159c80 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.856+0000 7f45227fc700 1 -- 172.21.15.77:0/2446140323 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4524019040 con 0x7f452c159c80 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.857+0000 7f45227fc700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 0x7f451006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.857+0000 7f45227fc700 1 -- 172.21.15.77:0/2446140323 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4524089940 con 0x7f452c159c80 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.857+0000 7f4531f99700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 0x7f451006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.866 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.857+0000 7f4531f99700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 0x7f451006d190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45280048d0 tx=0x7f4528009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.859+0000 7f4540382b80 1 Processor -- start 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.859+0000 7f4540382b80 1 -- start start 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3a8d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ab7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3991c60 con 0x5628c3ad5160 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c39e49f0 con 0x5628c3ac8c40 2023-07-04T18:52:30.867 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c39dde80 con 0x5628c3ad4d90 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46110/0 (socket says 172.21.15.77:46110) 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 -- 172.21.15.77:0/3706350675 learned_addr learned my addr 172.21.15.77:0/3706350675 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531f99700 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3a8d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f453279a700 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ab7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 -- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 msgr2=0x5628c3a8d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.868 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3a8d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 -- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 msgr2=0x5628c3ab7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ab7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 -- 172.21.15.77:0/3706350675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c380bd90 con 0x5628c3ad5160 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f4531798700 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f451c009b20 tx=0x7f451c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce3d403ef5abb09 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f450f7fe700 1 -- 172.21.15.77:0/3706350675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c011960 con 0x5628c3ad5160 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f450f7fe700 1 -- 172.21.15.77:0/3706350675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c011ac0 con 0x5628c3ad5160 2023-07-04T18:52:30.869 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.860+0000 7f450f7fe700 1 -- 172.21.15.77:0/3706350675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c011e50 con 0x5628c3ad5160 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 -- 172.21.15.77:0/3706350675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 msgr2=0x5628c3ab7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f451c009b20 tx=0x7f451c009e30 comp rx=0 tx=0).stop 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 -- 172.21.15.77:0/3706350675 shutdown_connections 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ab7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3a8d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 --2- 172.21.15.77:0/3706350675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ab7030 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 -- 172.21.15.77:0/3706350675 >> 172.21.15.77:0/3706350675 conn(0x5628c3a81a90 msgr2=0x5628c3a87010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.870 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 -- 172.21.15.77:0/3706350675 shutdown_connections 2023-07-04T18:52:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.861+0000 7f4540382b80 1 -- 172.21.15.77:0/3706350675 wait complete. 2023-07-04T18:52:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.862+0000 7f4540382b80 1 -- start start 2023-07-04T18:52:30.871 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.862+0000 7f4540382b80 1 Processor -- start 2023-07-04T18:52:30.874 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3aa2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad5160 0x5628c3aa1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c397da70 con 0x5628c3ad4d90 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c397d770 con 0x5628c3ac8c40 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c38618d0 con 0x5628c3ad5160 2023-07-04T18:52:30.875 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 10 client.0 _ll_register_callbacks cb 0x5628c397b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46116/0 (socket says 172.21.15.77:46116) 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 -- 172.21.15.77:0/3417141341 learned_addr learned my addr 172.21.15.77:0/3417141341 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f453279a700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad5160 0x5628c3aa1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531f99700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3aa2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 -- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad5160 msgr2=0x5628c3aa1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad5160 0x5628c3aa1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.876 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 -- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 msgr2=0x5628c3aa2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3aa2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 -- 172.21.15.77:0/3417141341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c0097d0 con 0x5628c3ad4d90 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4531798700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f451c009af0 tx=0x7f451c009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18eddd416cfad4a8 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f450cff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c011960 con 0x5628c3ad4d90 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f451c014b80 con 0x5628c3ad4d90 2023-07-04T18:52:30.877 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628c3a7f0a0 con 0x5628c3ad4d90 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f451c014d40 con 0x5628c3ad4d90 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 10 client.5346 fetch_fsmap learned FSMap version 16 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.863+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5628c37f4c10 con 0x5628c3ad4d90 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5628c3a7f0a0 con 0x5628c3ad4d90 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f451c01b020 con 0x5628c3ad4d90 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 10 client.5346 fetch_fsmap finished waiting for FSMap version 16 2023-07-04T18:52:30.878 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 10 client.5346 resolve_mds: resolved name 'd' to daemon mds.d 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 20 client.5346 populate_metadata read hostname 'smithi077' 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3a9f530 0x5628c3a9f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 4 client.5346 mds_command: new command op to 5136 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 --> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628c39d6e60 con 0x5628c3a9f530 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4531f99700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3a9f530 0x5628c3a9f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f450cff9700 10 client.5346 ms_handle_connect on v2:172.21.15.82:6835/744114579 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.864+0000 7f4531f99700 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3a9f530 0x5628c3a9f900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20350c1cf488ba12 server_cookie=b11e137716aa4844 in_seq=0 out_seq=0 2023-07-04T18:52:30.879 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f450cff9700 1 -- 172.21.15.77:0/3417141341 <== mds.0 v2:172.21.15.82:6835/744114579 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5628c39d6e60 con 0x5628c3a9f530 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f450cff9700 10 client.5346 handle_command_reply: tid=0 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 client.5346 shutdown 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 msgr2=0x5628c3a87320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f451c009af0 tx=0x7f451c009210 comp rx=0 tx=0).stop 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 shutdown_connections 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3a9f530 0x5628c3a9f900 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.880 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad5160 0x5628c3aa1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f450cff9700 0 client.5346 ms_handle_reset on v2:172.21.15.82:6835/744114579 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3aa2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 --2- 172.21.15.77:0/3417141341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3a87320 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 >> 172.21.15.77:0/3417141341 conn(0x5628c3a81a90 msgr2=0x5628c3ad2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 shutdown_connections 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 1 -- 172.21.15.77:0/3417141341 wait complete. 2023-07-04T18:52:30.881 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.865+0000 7f4540382b80 20 client.5346 trim_cache size 0 max 16384 2023-07-04T18:52:30.884 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.883+0000 7f4540382b80 1 Processor -- start 2023-07-04T18:52:30.885 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.883+0000 7f4540382b80 1 -- start start 2023-07-04T18:52:30.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ae3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.886 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3ad5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3a883c0 con 0x5628c3ad5160 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3aa2320 con 0x5628c3ac8c40 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c3aa1a10 con 0x5628c3ad4d90 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46126/0 (socket says 172.21.15.77:46126) 2023-07-04T18:52:30.887 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 -- 172.21.15.77:0/947173995 learned_addr learned my addr 172.21.15.77:0/947173995 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4531798700 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3ad5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f4531f99700 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ae3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 -- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 msgr2=0x5628c3ad5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3ad5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 -- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 msgr2=0x5628c3ae3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ae3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 -- 172.21.15.77:0/947173995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c3a9f000 con 0x5628c3ad5160 2023-07-04T18:52:30.888 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f453279a700 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f45240586d0 tx=0x7f4524005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7817b7e8da2db9c server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f450ffff700 1 -- 172.21.15.77:0/947173995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524089050 con 0x5628c3ad5160 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f450ffff700 1 -- 172.21.15.77:0/947173995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4524060a90 con 0x5628c3ad5160 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.884+0000 7f450ffff700 1 -- 172.21.15.77:0/947173995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524001f10 con 0x5628c3ad5160 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- 172.21.15.77:0/947173995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 msgr2=0x5628c3ad5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f45240586d0 tx=0x7f4524005040 comp rx=0 tx=0).stop 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- 172.21.15.77:0/947173995 shutdown_connections 2023-07-04T18:52:30.889 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad5160 0x5628c3ad5a70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ac8c40 0x5628c3ae3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 --2- 172.21.15.77:0/947173995 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ad4d90 0x5628c3ad5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- 172.21.15.77:0/947173995 >> 172.21.15.77:0/947173995 conn(0x5628c3a81a90 msgr2=0x5628c3a87130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- 172.21.15.77:0/947173995 shutdown_connections 2023-07-04T18:52:30.890 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- 172.21.15.77:0/947173995 wait complete. 2023-07-04T18:52:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 -- start start 2023-07-04T18:52:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.885+0000 7f4540382b80 1 Processor -- start 2023-07-04T18:52:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ac8c40 0x5628c3a86120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.893 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ad5160 0x5628c3b22450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3ad5f70 con 0x5628c3ad4d90 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3ab7c40 con 0x5628c3ad5160 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c3a86660 con 0x5628c3ac8c40 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 10 client.0 _ll_register_callbacks cb 0x5628c397b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.894 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46130/0 (socket says 172.21.15.77:46130) 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 -- 172.21.15.77:0/2481462776 learned_addr learned my addr 172.21.15.77:0/2481462776 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531f99700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ac8c40 0x5628c3a86120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f453279a700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ad5160 0x5628c3b22450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 -- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ac8c40 msgr2=0x5628c3a86120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ac8c40 0x5628c3a86120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 -- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ad5160 msgr2=0x5628c3b22450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ad5160 0x5628c3b22450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.895 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 -- 172.21.15.77:0/2481462776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c0b3850 con 0x5628c3ad4d90 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4531798700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f451c006dc0 tx=0x7f451c006ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bf45a261ed50dd server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f450cff9700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c039070 con 0x5628c3ad4d90 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f451c000e40 con 0x5628c3ad4d90 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.887+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f451c011ac0 con 0x5628c3ad4d90 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f451c011960 con 0x5628c3ad4d90 2023-07-04T18:52:30.896 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 10 client.5352 fetch_fsmap learned FSMap version 16 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5628c3832200 con 0x5628c3ad4d90 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c011ac0 con 0x5628c3ad4d90 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f451c0112c0 con 0x5628c3ad4d90 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 10 client.5352 fetch_fsmap finished waiting for FSMap version 16 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 10 client.5352 resolve_mds: resolved name 'd' to daemon mds.d 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 20 client.5352 populate_metadata read hostname 'smithi077' 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3ad2b60 0x5628c3ad2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:30.897 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 4 client.5352 mds_command: new command op to 5136 tid=0[{"prefix": "status"}] 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 --> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5628c3999000 con 0x5628c3ad2b60 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4531f99700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3ad2b60 0x5628c3ad2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f450cff9700 10 client.5352 ms_handle_connect on v2:172.21.15.82:6835/744114579 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.888+0000 7f4531f99700 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3ad2b60 0x5628c3ad2f30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=274fd45336ba849d server_cookie=aa56de07b0bcc095 in_seq=0 out_seq=0 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f450cff9700 1 -- 172.21.15.77:0/2481462776 <== mds.0 v2:172.21.15.82:6835/744114579 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5628c3999000 con 0x5628c3ad2b60 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f450cff9700 10 client.5352 handle_command_reply: tid=0 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 client.5352 shutdown 2023-07-04T18:52:30.898 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 msgr2=0x5628c3b21f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f451c006dc0 tx=0x7f451c006ab0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 shutdown_connections 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:6835/744114579,v1:172.21.15.82:6837/744114579] conn(0x5628c3ad2b60 0x5628c3ad2f30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5628c3ad5160 0x5628c3b22450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x5628c3ac8c40 0x5628c3a86120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f450cff9700 0 client.5352 ms_handle_reset on v2:172.21.15.82:6835/744114579 2023-07-04T18:52:30.899 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 --2- 172.21.15.77:0/2481462776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5628c3ad4d90 0x5628c3b21f10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 >> 172.21.15.77:0/2481462776 conn(0x5628c3a81a90 msgr2=0x5628c3ae3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 shutdown_connections 2023-07-04T18:52:30.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 1 -- 172.21.15.77:0/2481462776 wait complete. 2023-07-04T18:52:30.900 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f4540382b80 20 client.5352 trim_cache size 0 max 16384 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5", 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 5136, 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2023-07-04T18:52:30.901 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 10.416047449000001, 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 16, 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 37, 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 37, 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 19.313141616999999 2023-07-04T18:52:30.902 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-07-04T18:52:30.903 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 msgr2=0x7f451006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 0x7f451006d190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45280048d0 tx=0x7f4528009040 comp rx=0 tx=0).stop 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 msgr2=0x7f452c1f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.889+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4524015040 tx=0x7f45240014a0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 shutdown_connections 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f451006ace0 0x7f451006d190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.904 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f452c159c80 0x7f452c1f5b80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f452c0b0960 0x7f452c158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 --2- 172.21.15.77:0/2446140323 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f452c159810 0x7f452c1f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:30.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 >> 172.21.15.77:0/2446140323 conn(0x7f452c06bba0 msgr2=0x7f452c0cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:30.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 shutdown_connections 2023-07-04T18:52:30.905 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:30.890+0000 7f45341fd700 1 -- 172.21.15.77:0/2446140323 wait complete. 2023-07-04T18:52:30.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-07-04T18:52:31.035 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.024+0000 7f455afd4700 1 Processor -- start 2023-07-04T18:52:31.035 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.024+0000 7f455afd4700 1 -- start start 2023-07-04T18:52:31.036 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f455afd4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f455415c640 0x7f45540cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.036 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f455afd4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4554070f80 0x7f45540cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.036 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f455afd4700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45540717f0 con 0x7f455415ca10 2023-07-04T18:52:31.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f455afd4700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45540cd290 con 0x7f455415c640 2023-07-04T18:52:31.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f455afd4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45540ccfb0 con 0x7f4554070f80 2023-07-04T18:52:31.037 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f455415c640 0x7f45540cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f455415c640 0x7f45540cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46136/0 (socket says 172.21.15.77:46136) 2023-07-04T18:52:31.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 -- 172.21.15.77:0/3730411071 learned_addr learned my addr 172.21.15.77:0/3730411071 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:31.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4559571700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4554070f80 0x7f45540cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.038 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 -- 172.21.15.77:0/3730411071 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4554070f80 msgr2=0x7f45540cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:31.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4554070f80 0x7f45540cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 -- 172.21.15.77:0/3730411071 >> v1:172.21.15.82:6789/0 conn(0x7f455415ca10 legacy=0x7f4554070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 -- 172.21.15.77:0/3730411071 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45540ce640 con 0x7f455415c640 2023-07-04T18:52:31.039 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.025+0000 7f4558d70700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415c640 0x7f45540cf120 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45480098f0 tx=0x7f4548009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc2bb08b6038102 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:31.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f45537fe700 1 -- 172.21.15.77:0/3730411071 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454800b8a0 con 0x7f455415c640 2023-07-04T18:52:31.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f45537fe700 1 -- 172.21.15.77:0/3730411071 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454800ba00 con 0x7f455415c640 2023-07-04T18:52:31.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f45537fe700 1 -- 172.21.15.77:0/3730411071 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454800bd90 con 0x7f455415c640 2023-07-04T18:52:31.040 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 -- 172.21.15.77:0/3730411071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415c640 msgr2=0x7f45540cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415c640 0x7f45540cf120 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f45480098f0 tx=0x7f4548009c00 comp rx=0 tx=0).stop 2023-07-04T18:52:31.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 -- 172.21.15.77:0/3730411071 shutdown_connections 2023-07-04T18:52:31.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f4554070f80 0x7f45540cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.041 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 --2- 172.21.15.77:0/3730411071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415c640 0x7f45540cf120 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 -- 172.21.15.77:0/3730411071 >> 172.21.15.77:0/3730411071 conn(0x7f455406bba0 msgr2=0x7f45540b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:31.042 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 -- 172.21.15.77:0/3730411071 shutdown_connections 2023-07-04T18:52:31.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.026+0000 7f455afd4700 1 -- 172.21.15.77:0/3730411071 wait complete. 2023-07-04T18:52:31.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.027+0000 7f455afd4700 1 Processor -- start 2023-07-04T18:52:31.043 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.027+0000 7f455afd4700 1 -- start start 2023-07-04T18:52:31.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.028+0000 7f455afd4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 0x7f4554153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.028+0000 7f455afd4700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f455415c640 0x7f45540cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.028+0000 7f455afd4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.044 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.028+0000 7f455afd4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45540b01c0 con 0x7f455415ca10 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f455afd4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45540b05e0 con 0x7f4554070f80 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f455afd4700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45540af540 con 0x7f455415c640 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46144/0 (socket says 172.21.15.77:46144) 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 -- 172.21.15.77:0/4104303892 learned_addr learned my addr 172.21.15.77:0/4104303892 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:31.045 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4558d70700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 0x7f4554153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4553fff700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f455415c640 0x7f45540cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 -- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f455415c640 msgr2=0x7f45540cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f455415c640 0x7f45540cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 -- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 msgr2=0x7f4554153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 0x7f4554153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 -- 172.21.15.77:0/4104303892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4548009550 con 0x7f455415ca10 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4558d70700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 0x7f4554153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:31.046 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f4559571700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f454c009fd0 tx=0x7f454c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:31.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c00ede0 con 0x7f455415ca10 2023-07-04T18:52:31.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c014420 con 0x7f455415ca10 2023-07-04T18:52:31.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45540af820 con 0x7f455415ca10 2023-07-04T18:52:31.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c0147d0 con 0x7f455415ca10 2023-07-04T18:52:31.047 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.029+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45540afde0 con 0x7f455415ca10 2023-07-04T18:52:31.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.030+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45540b0300 con 0x7f455415ca10 2023-07-04T18:52:31.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.031+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f454c014930 con 0x7f455415ca10 2023-07-04T18:52:31.048 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.031+0000 7f45517fa700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 0x7f453806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:31.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.031+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f454c089b00 con 0x7f455415ca10 2023-07-04T18:52:31.049 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.031+0000 7f4558d70700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 0x7f453806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:31.050 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.032+0000 7f4558d70700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 0x7f453806d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4548014040 tx=0x7f4548008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:31.050 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.035+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f45540b0300 con 0x7f455415ca10 2023-07-04T18:52:31.268 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.267+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f45540b0300 con 0x7f455415ca10 2023-07-04T18:52:31.269 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.267+0000 7f45517fa700 1 -- 172.21.15.77:0/4104303892 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f45540b0300 con 0x7f455415ca10 2023-07-04T18:52:31.269 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-04T18:52:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 msgr2=0x7f453806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 0x7f453806d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4548014040 tx=0x7f4548008030 comp rx=0 tx=0).stop 2023-07-04T18:52:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 msgr2=0x7f45540cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:31.271 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f454c009fd0 tx=0x7f454c00e3c0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 shutdown_connections 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f453806ace0 0x7f453806d190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f455415ca10 0x7f45540cbb20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4554070f80 0x7f4554153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 --2- 172.21.15.77:0/4104303892 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f455415c640 0x7f45540cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:31.272 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 >> 172.21.15.77:0/4104303892 conn(0x7f455406bba0 msgr2=0x7f45540ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:31.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.269+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 shutdown_connections 2023-07-04T18:52:31.273 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:31.270+0000 7f455afd4700 1 -- 172.21.15.77:0/4104303892 wait complete. 2023-07-04T18:52:31.283 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-04T18:52:31.283 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-04T18:52:31.284 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2023-07-04T18:52:31.284 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-04T18:52:31.284 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-04T18:52:31.284 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-04T18:52:31.285 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-07-04T18:52:31.285 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-04T18:52:31.286 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-04T18:52:31.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.286 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2023-07-04T18:52:31.303 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.304 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-04T18:52:31.305 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2023-07-04T18:52:31.306 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42118sec preferred_lft 42118sec 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::8178:5374:91c4:639d/64 scope link noprefixroute 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 5e:ca:c7:f8:96:fc brd ff:ff:ff:ff:ff:ff 2023-07-04T18:52:31.307 INFO:teuthology.orchestra.run.smithi077.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-07-04T18:52:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ac9c:d0ff:fedf:fce1/64 scope link 2023-07-04T18:52:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.308 INFO:teuthology.orchestra.run.smithi077.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-07-04T18:52:31.308 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 5e:ca:c7:f8:96:fc brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-04T18:52:31.309 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::5cca:c7ff:fef8:96fc/64 scope link 2023-07-04T18:52:31.309 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.309 INFO:teuthology.orchestra.run.smithi077.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-07-04T18:52:31.309 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether a2:bf:01:2a:0d:6a brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-07-04T18:52:31.310 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::a0bf:1ff:fe2a:d6a/64 scope link 2023-07-04T18:52:31.310 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2023-07-04T18:52:31.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.310 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-04T18:52:31.368 INFO:teuthology.orchestra.run.smithi077.stdout:1 2023-07-04T18:52:31.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.370 DEBUG:teuthology.orchestra.run.smithi077:> route 2023-07-04T18:52:31.429 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2023-07-04T18:52:31.430 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-04T18:52:31.430 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-04T18:52:31.430 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-04T18:52:31.430 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-04T18:52:31.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.431 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2023-07-04T18:52:31.432 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2023-07-04T18:52:31.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-04T18:52:31.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-04T18:52:31.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-04T18:52:31.433 DEBUG:teuthology.orchestra.run.smithi077:> ') 2023-07-04T18:52:31.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.534 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2023-07-04T18:52:31.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-07-04T18:52:31.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-04T18:52:31.552 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-04T18:52:31.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2023-07-04T18:52:31.618 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:52:31.619 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-04T18:52:31.675 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-04T18:52:31.676 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-04T18:52:31.676 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-04T18:52:31.732 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-07-04T18:52:31.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:52:31.817 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:52:31.817 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:52:31.841 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:52:31.842 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:52:31.842 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:52:31.842 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:52:31.842 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:52:31.843 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:52:31.843 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:52:31.843 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:52:31.872 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-07-04T18:52:31.872 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-07-04T18:52:31.872 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.82:3300,172.21.15.82:3301" 2023-07-04T18:52:31.873 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with new device syntax: 0@5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5.cephfs=/ 2023-07-04T18:52:31.873 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.77:3300/172.21.15.82:3300/172.21.15.82:3301" will pass to kernel 2023-07-04T18:52:31.873 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: v2 mount support check returned 2 2023-07-04T18:52:31.873 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-04T18:52:31.873 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2023-07-04T18:52:31.874 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790" 2023-07-04T18:52:31.874 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.82:6789,172.21.15.82:6790:/ 2023-07-04T18:52:31.874 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5" will pass to kernel 2023-07-04T18:52:31.883 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-04T18:52:31.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.884 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-04T18:52:31.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:31.918 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-04T18:52:32.004 INFO:teuthology.orchestra.run.smithi077.stdout:client5361 2023-07-04T18:52:32.004 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:52:32.005 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T18:52:32.021 DEBUG:teuthology.orchestra.run.smithi077:> mktemp -d --suffix=xfstests-dev 2023-07-04T18:52:32.078 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.pA346HXQMhxfstests-dev 2023-07-04T18:52:32.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:32.079 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo chmod a+rx /tmp/tmp.pA346HXQMhxfstests-dev) 2023-07-04T18:52:32.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:32.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec git clone https://git.ceph.com/xfstests-dev.git --depth 1 /tmp/tmp.pA346HXQMhxfstests-dev) 2023-07-04T18:52:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:Cloning into '/tmp/tmp.pA346HXQMhxfstests-dev'... 2023-07-04T18:52:32.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:32.911 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test) 2023-07-04T18:52:32.975 DEBUG:teuthology.orchestra.run.smithi077:> mktemp -d --suffix=test 2023-07-04T18:52:33.032 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.ujtNGq2JOmtest 2023-07-04T18:52:33.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:52:33.034 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir scratch) 2023-07-04T18:52:33.093 DEBUG:teuthology.orchestra.run.smithi077:> mktemp -d --suffix=scratch 2023-07-04T18:52:33.150 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:52:33.151 INFO:tasks.cephfs.xfstests_dev:distro and version detected is "rhel" and "8.6". 2023-07-04T18:52:33.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y acl attr automake bc dbench dump e2fsprogs fio gawk gcc indent libtool lvm2 make psmisc quota sed xfsdump xfsprogs libacl-devel libattr-devel libaio-devel libuuid-devel xfsprogs-devel python2 sqlite xfsprogs-qa-devel 2023-07-04T18:52:33.544 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2023-07-04T18:52:34.368 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:07:04 ago on Tue 04 Jul 2023 06:45:30 PM UTC. 2023-07-04T18:52:35.656 INFO:teuthology.orchestra.run.smithi077.stdout:Package acl-2.2.53-1.el8.x86_64 is already installed. 2023-07-04T18:52:35.657 INFO:teuthology.orchestra.run.smithi077.stdout:Package attr-2.4.48-3.el8.x86_64 is already installed. 2023-07-04T18:52:35.659 INFO:teuthology.orchestra.run.smithi077.stdout:Package automake-1.16.1-7.el8.noarch is already installed. 2023-07-04T18:52:35.660 INFO:teuthology.orchestra.run.smithi077.stdout:Package bc-1.07.1-5.el8.x86_64 is already installed. 2023-07-04T18:52:35.661 INFO:teuthology.orchestra.run.smithi077.stdout:Package dbench-4.0-20.el8.x86_64 is already installed. 2023-07-04T18:52:35.663 INFO:teuthology.orchestra.run.smithi077.stdout:Package e2fsprogs-1.45.6-4.el8.x86_64 is already installed. 2023-07-04T18:52:35.665 INFO:teuthology.orchestra.run.smithi077.stdout:Package gawk-4.2.1-4.el8.x86_64 is already installed. 2023-07-04T18:52:35.666 INFO:teuthology.orchestra.run.smithi077.stdout:Package gcc-8.5.0-10.1.el8_6.x86_64 is already installed. 2023-07-04T18:52:35.668 INFO:teuthology.orchestra.run.smithi077.stdout:Package libtool-2.4.6-25.el8.x86_64 is already installed. 2023-07-04T18:52:35.669 INFO:teuthology.orchestra.run.smithi077.stdout:Package lvm2-8:2.03.14-3.el8_6.2.x86_64 is already installed. 2023-07-04T18:52:35.670 INFO:teuthology.orchestra.run.smithi077.stdout:Package make-1:4.2.1-11.el8.x86_64 is already installed. 2023-07-04T18:52:35.671 INFO:teuthology.orchestra.run.smithi077.stdout:Package psmisc-23.1-5.el8.x86_64 is already installed. 2023-07-04T18:52:35.672 INFO:teuthology.orchestra.run.smithi077.stdout:Package quota-1:4.04-14.el8.x86_64 is already installed. 2023-07-04T18:52:35.673 INFO:teuthology.orchestra.run.smithi077.stdout:Package sed-4.5-5.el8.x86_64 is already installed. 2023-07-04T18:52:35.675 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-04T18:52:35.676 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-04T18:52:35.677 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-04T18:52:35.678 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-04T18:52:35.679 INFO:teuthology.orchestra.run.smithi077.stdout:Package libaio-devel-0.3.112-1.el8.x86_64 is already installed. 2023-07-04T18:52:35.680 INFO:teuthology.orchestra.run.smithi077.stdout:Package libuuid-devel-2.32.1-35.el8.x86_64 is already installed. 2023-07-04T18:52:35.681 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-devel-5.0.0-10.el8.x86_64 is already installed. 2023-07-04T18:52:35.685 INFO:teuthology.orchestra.run.smithi077.stdout:Package sqlite-3.26.0-15.el8.x86_64 is already installed. 2023-07-04T18:52:35.688 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-04T18:52:35.801 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-07-04T18:52:35.803 INFO:teuthology.orchestra.run.smithi077.stdout:==================================================================================================================== 2023-07-04T18:52:35.804 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-07-04T18:52:35.804 INFO:teuthology.orchestra.run.smithi077.stdout:==================================================================================================================== 2023-07-04T18:52:35.804 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-07-04T18:52:35.804 INFO:teuthology.orchestra.run.smithi077.stdout: dump x86_64 1:0.4-0.36.b46.el8 rhel-8-for-x86_64-baseos-rpms 164 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: fio x86_64 3.19-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: indent x86_64 2.2.13-2.el8 epel 187 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: python2 x86_64 2.7.18-10.module+el8.6.0+14191+7fdd52cd rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: sqlite x86_64 3.26.0-16.el8_6 rhel-8-for-x86_64-baseos-rpms 668 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: sqlite-libs x86_64 3.26.0-16.el8_6 rhel-8-for-x86_64-baseos-rpms 581 k 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-07-04T18:52:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: daxctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 42 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemblk x86_64 1.6.1-1.el8 rhel-8-for-x86_64-appstream-rpms 92 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: ndctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 79 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: python2-libs x86_64 2.7.18-10.module+el8.6.0+14191+7fdd52cd rhel-8-for-x86_64-appstream-rpms 6.0 M 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: python2-pip-wheel noarch 9.0.3-19.module+el8.6.0+13001+ad200bd9 rhel-8-for-x86_64-appstream-rpms 892 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: python2-setuptools-wheel noarch 39.0.1-13.module+el8.4.0+9442+27d0e81c rhel-8-for-x86_64-appstream-rpms 287 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: python2-wheel-wheel noarch 1:0.31.1-3.module+el8.5.0+12203+77770ab7 rhel-8-for-x86_64-appstream-rpms 45 k 2023-07-04T18:52:35.806 INFO:teuthology.orchestra.run.smithi077.stdout: rmt x86_64 2:1.5.3-13.el8 rhel-8-for-x86_64-baseos-rpms 51 k 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout: python2-pip noarch 9.0.3-19.module+el8.6.0+13001+ad200bd9 rhel-8-for-x86_64-appstream-rpms 1.6 M 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout: python2-setuptools noarch 39.0.1-13.module+el8.4.0+9442+27d0e81c rhel-8-for-x86_64-appstream-rpms 642 k 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout: python27 2.7 2023-07-04T18:52:35.807 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:35.808 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-07-04T18:52:35.808 INFO:teuthology.orchestra.run.smithi077.stdout:==================================================================================================================== 2023-07-04T18:52:35.808 INFO:teuthology.orchestra.run.smithi077.stdout:Install 14 Packages 2023-07-04T18:52:35.809 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2023-07-04T18:52:35.809 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:35.809 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 12 M 2023-07-04T18:52:35.809 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-07-04T18:52:35.971 INFO:teuthology.orchestra.run.smithi077.stdout:(1/16): dump-0.4-0.36.b46.el8.x86_64.rpm 1.7 MB/s | 164 kB 00:00 2023-07-04T18:52:35.983 INFO:teuthology.orchestra.run.smithi077.stdout:(2/16): rmt-1.5.3-13.el8.x86_64.rpm 461 kB/s | 51 kB 00:00 2023-07-04T18:52:36.009 INFO:teuthology.orchestra.run.smithi077.stdout:(3/16): daxctl-libs-71.1-3.el8.x86_64.rpm 993 kB/s | 42 kB 00:00 2023-07-04T18:52:36.034 INFO:teuthology.orchestra.run.smithi077.stdout:(4/16): ndctl-libs-71.1-3.el8.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2023-07-04T18:52:36.060 INFO:teuthology.orchestra.run.smithi077.stdout:(5/16): python2-setuptools-wheel-39.0.1-13.modu 5.6 MB/s | 287 kB 00:00 2023-07-04T18:52:36.093 INFO:teuthology.orchestra.run.smithi077.stdout:(6/16): python2-setuptools-39.0.1-13.module+el8 11 MB/s | 642 kB 00:00 2023-07-04T18:52:36.123 INFO:teuthology.orchestra.run.smithi077.stdout:(7/16): indent-2.2.13-2.el8.x86_64.rpm 756 kB/s | 187 kB 00:00 2023-07-04T18:52:36.144 INFO:teuthology.orchestra.run.smithi077.stdout:(8/16): python2-wheel-wheel-0.31.1-3.module+el8 532 kB/s | 45 kB 00:00 2023-07-04T18:52:36.178 INFO:teuthology.orchestra.run.smithi077.stdout:(9/16): fio-3.19-3.el8.x86_64.rpm 6.3 MB/s | 538 kB 00:00 2023-07-04T18:52:36.203 INFO:teuthology.orchestra.run.smithi077.stdout:(10/16): libpmemblk-1.6.1-1.el8.x86_64.rpm 1.1 MB/s | 92 kB 00:00 2023-07-04T18:52:36.228 INFO:teuthology.orchestra.run.smithi077.stdout:(11/16): python2-2.7.18-10.module+el8.6.0+14191 1.4 MB/s | 110 kB 00:00 2023-07-04T18:52:36.287 INFO:teuthology.orchestra.run.smithi077.stdout:(12/16): python2-pip-9.0.3-19.module+el8.6.0+13 20 MB/s | 1.6 MB 00:00 2023-07-04T18:52:36.341 INFO:teuthology.orchestra.run.smithi077.stdout:(13/16): python2-pip-wheel-9.0.3-19.module+el8. 8.0 MB/s | 892 kB 00:00 2023-07-04T18:52:36.379 INFO:teuthology.orchestra.run.smithi077.stdout:(14/16): sqlite-3.26.0-16.el8_6.x86_64.rpm 7.1 MB/s | 668 kB 00:00 2023-07-04T18:52:36.415 INFO:teuthology.orchestra.run.smithi077.stdout:(15/16): sqlite-libs-3.26.0-16.el8_6.x86_64.rpm 7.6 MB/s | 581 kB 00:00 2023-07-04T18:52:36.488 INFO:teuthology.orchestra.run.smithi077.stdout:(16/16): python2-libs-2.7.18-10.module+el8.6.0+ 19 MB/s | 6.0 MB 00:00 2023-07-04T18:52:36.489 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-07-04T18:52:36.489 INFO:teuthology.orchestra.run.smithi077.stdout:Total 17 MB/s | 12 MB 00:00 2023-07-04T18:52:36.698 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-07-04T18:52:36.738 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-07-04T18:52:36.739 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-07-04T18:52:36.924 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-07-04T18:52:36.927 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-07-04T18:52:37.652 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-07-04T18:52:37.865 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : sqlite-libs-3.26.0-16.el8_6.x86_64 1/18 2023-07-04T18:52:38.009 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : daxctl-libs-71.1-3.el8.x86_64 2/18 2023-07-04T18:52:38.084 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: daxctl-libs-71.1-3.el8.x86_64 2/18 2023-07-04T18:52:38.309 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ndctl-libs-71.1-3.el8.x86_64 3/18 2023-07-04T18:52:38.377 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ndctl-libs-71.1-3.el8.x86_64 3/18 2023-07-04T18:52:38.551 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemblk-1.6.1-1.el8.x86_64 4/18 2023-07-04T18:52:38.649 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libpmemblk-1.6.1-1.el8.x86_64 4/18 2023-07-04T18:52:38.788 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-pip-wheel-9.0.3-19.module+el8.6.0+13001+ad 5/18 2023-07-04T18:52:39.062 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-setuptools-wheel-39.0.1-13.module+el8.4.0+ 6/18 2023-07-04T18:52:39.824 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-setuptools-39.0.1-13.module+el8.4.0+9442+2 7/18 2023-07-04T18:52:40.089 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-pip-9.0.3-19.module+el8.6.0+13001+ad200bd9 8/18 2023-07-04T18:52:40.254 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-2.7.18-10.module+el8.6.0+14191+7fdd52cd.x8 9/18 2023-07-04T18:52:41.375 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python2-2.7.18-10.module+el8.6.0+14191+7fdd52cd.x8 9/18 2023-07-04T18:52:41.690 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-libs-2.7.18-10.module+el8.6.0+14191+7fdd52 10/18 2023-07-04T18:52:41.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rmt-2:1.5.3-13.el8.x86_64 11/18 2023-07-04T18:52:42.200 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : dump-1:0.4-0.36.b46.el8.x86_64 12/18 2023-07-04T18:52:42.488 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fio-3.19-3.el8.x86_64 13/18 2023-07-04T18:52:42.670 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : sqlite-3.26.0-16.el8_6.x86_64 14/18 2023-07-04T18:52:42.863 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python2-wheel-wheel-1:0.31.1-3.module+el8.5.0+1220 15/18 2023-07-04T18:52:43.067 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : indent-2.2.13-2.el8.x86_64 16/18 2023-07-04T18:52:43.209 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : sqlite-3.26.0-15.el8.x86_64 17/18 2023-07-04T18:52:43.382 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : sqlite-libs-3.26.0-15.el8.x86_64 18/18 2023-07-04T18:52:43.660 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: sqlite-libs-3.26.0-15.el8.x86_64 18/18 2023-07-04T18:52:43.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : indent-2.2.13-2.el8.x86_64 1/18 2023-07-04T18:52:43.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : dump-1:0.4-0.36.b46.el8.x86_64 2/18 2023-07-04T18:52:43.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rmt-2:1.5.3-13.el8.x86_64 3/18 2023-07-04T18:52:43.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : daxctl-libs-71.1-3.el8.x86_64 4/18 2023-07-04T18:52:43.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ndctl-libs-71.1-3.el8.x86_64 5/18 2023-07-04T18:52:43.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-setuptools-wheel-39.0.1-13.module+el8.4.0+ 6/18 2023-07-04T18:52:43.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-setuptools-39.0.1-13.module+el8.4.0+9442+2 7/18 2023-07-04T18:52:43.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-wheel-wheel-1:0.31.1-3.module+el8.5.0+1220 8/18 2023-07-04T18:52:43.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fio-3.19-3.el8.x86_64 9/18 2023-07-04T18:52:43.664 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemblk-1.6.1-1.el8.x86_64 10/18 2023-07-04T18:52:43.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-2.7.18-10.module+el8.6.0+14191+7fdd52cd.x8 11/18 2023-07-04T18:52:43.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-libs-2.7.18-10.module+el8.6.0+14191+7fdd52 12/18 2023-07-04T18:52:43.666 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-pip-9.0.3-19.module+el8.6.0+13001+ad200bd9 13/18 2023-07-04T18:52:43.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python2-pip-wheel-9.0.3-19.module+el8.6.0+13001+ad 14/18 2023-07-04T18:52:43.668 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : sqlite-3.26.0-16.el8_6.x86_64 15/18 2023-07-04T18:52:43.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : sqlite-3.26.0-15.el8.x86_64 16/18 2023-07-04T18:52:43.669 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : sqlite-libs-3.26.0-16.el8_6.x86_64 17/18 2023-07-04T18:52:44.910 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : sqlite-libs-3.26.0-15.el8.x86_64 18/18 2023-07-04T18:52:44.911 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:07:14 ago on Tue 04 Jul 2023 06:45:30 PM UTC. 2023-07-04T18:52:47.668 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2023-07-04T18:52:47.859 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-07-04T18:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout: sqlite-3.26.0-16.el8_6.x86_64 sqlite-libs-3.26.0-16.el8_6.x86_64 2023-07-04T18:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-07-04T18:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout: daxctl-libs-71.1-3.el8.x86_64 2023-07-04T18:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout: dump-1:0.4-0.36.b46.el8.x86_64 2023-07-04T18:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout: fio-3.19-3.el8.x86_64 2023-07-04T18:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout: indent-2.2.13-2.el8.x86_64 2023-07-04T18:52:47.862 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemblk-1.6.1-1.el8.x86_64 2023-07-04T18:52:47.862 INFO:teuthology.orchestra.run.smithi077.stdout: ndctl-libs-71.1-3.el8.x86_64 2023-07-04T18:52:47.862 INFO:teuthology.orchestra.run.smithi077.stdout: python2-2.7.18-10.module+el8.6.0+14191+7fdd52cd.x86_64 2023-07-04T18:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout: python2-libs-2.7.18-10.module+el8.6.0+14191+7fdd52cd.x86_64 2023-07-04T18:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout: python2-pip-9.0.3-19.module+el8.6.0+13001+ad200bd9.noarch 2023-07-04T18:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout: python2-pip-wheel-9.0.3-19.module+el8.6.0+13001+ad200bd9.noarch 2023-07-04T18:52:47.864 INFO:teuthology.orchestra.run.smithi077.stdout: python2-setuptools-39.0.1-13.module+el8.4.0+9442+27d0e81c.noarch 2023-07-04T18:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout: python2-setuptools-wheel-39.0.1-13.module+el8.4.0+9442+27d0e81c.noarch 2023-07-04T18:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout: python2-wheel-wheel-1:0.31.1-3.module+el8.5.0+12203+77770ab7.noarch 2023-07-04T18:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout: rmt-2:1.5.3-13.el8.x86_64 2023-07-04T18:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:47.866 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-07-04T18:52:48.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo useradd -m fsgqa 2023-07-04T18:52:48.082 INFO:teuthology.orchestra.run.smithi077.stderr:useradd: user 'fsgqa' already exists 2023-07-04T18:52:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 9 2023-07-04T18:52:48.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo groupadd fsgqa 2023-07-04T18:52:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 9 2023-07-04T18:52:48.157 INFO:teuthology.orchestra.run.smithi077.stderr:groupadd: group 'fsgqa' already exists 2023-07-04T18:52:48.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo useradd fsgqa2 2023-07-04T18:52:49.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo useradd 123456-fsgqa 2023-07-04T18:52:49.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json mon dump 2023-07-04T18:52:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.112+0000 7f4052b4d700 1 Processor -- start 2023-07-04T18:52:50.114 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.112+0000 7f4052b4d700 1 -- start start 2023-07-04T18:52:50.115 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f4052b4d700 1 --2- >> v2:172.21.15.82:3300/0 conn(0x7f404c0cb940 0x7f404c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f4052b4d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0cdd70 con 0x7f404c158cc0 2023-07-04T18:52:50.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f4052b4d700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0ce050 con 0x7f404c06cbe0 2023-07-04T18:52:50.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f4052b4d700 1 -- --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404c0ce330 con 0x7f404c0cb940 2023-07-04T18:52:50.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f40508e9700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f404c158cc0 0x7f404c0710e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:53310/0 (socket says 172.21.15.77:53310) 2023-07-04T18:52:50.116 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.113+0000 7f40508e9700 1 -- 172.21.15.77:0/912859825 learned_addr learned my addr 172.21.15.77:0/912859825 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f40510ea700 1 --2- 172.21.15.77:0/912859825 >> v2:172.21.15.82:3300/0 conn(0x7f404c0cb940 0x7f404c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709252858 0 0) 0x7f404c0cdd70 con 0x7f404c158cc0 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c0034d0 con 0x7f404c158cc0 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f40510ea700 1 -- 172.21.15.77:0/912859825 >> v1:172.21.15.82:6789/0 conn(0x7f404c06cbe0 legacy=0x7f404c0cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1593569848 0 0) 0x7f403c0034d0 con 0x7f404c158cc0 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f40510ea700 1 -- 172.21.15.77:0/912859825 >> v1:172.21.15.77:6789/0 conn(0x7f404c158cc0 legacy=0x7f404c0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f40510ea700 1 -- 172.21.15.77:0/912859825 --> v2:172.21.15.82:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c0ce610 con 0x7f404c0cb940 2023-07-04T18:52:50.117 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f40510ea700 1 --2- 172.21.15.77:0/912859825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c06f550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4044009dc0 tx=0x7f404400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b207c5b1f06bff42 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404400e070 con 0x7f404c0cb940 2023-07-04T18:52:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4044003a20 con 0x7f404c0cb940 2023-07-04T18:52:50.118 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.114+0000 7f404b7fe700 1 -- 172.21.15.77:0/912859825 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040001670 con 0x7f404c0cb940 2023-07-04T18:52:50.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 -- 172.21.15.77:0/912859825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 msgr2=0x7f404c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.120 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 --2- 172.21.15.77:0/912859825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c06f550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4044009dc0 tx=0x7f404400b330 comp rx=0 tx=0).stop 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 -- 172.21.15.77:0/912859825 shutdown_connections 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 --2- 172.21.15.77:0/912859825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c06f550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 -- 172.21.15.77:0/912859825 >> 172.21.15.77:0/912859825 conn(0x7f404c06aad0 msgr2=0x7f404c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 -- 172.21.15.77:0/912859825 shutdown_connections 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.115+0000 7f4052b4d700 1 -- 172.21.15.77:0/912859825 wait complete. 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.116+0000 7f4052b4d700 1 Processor -- start 2023-07-04T18:52:50.121 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.116+0000 7f4052b4d700 1 -- start start 2023-07-04T18:52:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 0x7f404c149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.122 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c14a520 con 0x7f404c06cbe0 2023-07-04T18:52:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c14afa0 con 0x7f404c0cb940 2023-07-04T18:52:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f4052b4d700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404c14a700 con 0x7f404c158cc0 2023-07-04T18:52:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f40508e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.123 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f40508e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46178/0 (socket says 172.21.15.77:46178) 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f40508e9700 1 -- 172.21.15.77:0/4108571784 learned_addr learned my addr 172.21.15.77:0/4108571784 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f404bfff700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.117+0000 7f40510ea700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 0x7f404c149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 -- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 msgr2=0x7f404c149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 0x7f404c149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.124 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 -- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 msgr2=0x7f404c1f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c1f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 -- 172.21.15.77:0/4108571784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044009a70 con 0x7f404c06cbe0 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f404bfff700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c1f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40510ea700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 0x7f404c149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40508e9700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f404000c750 tx=0x7f404000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040004c80 con 0x7f404c06cbe0 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4040004e20 con 0x7f404c06cbe0 2023-07-04T18:52:50.125 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c14a9e0 con 0x7f404c06cbe0 2023-07-04T18:52:50.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040007310 con 0x7f404c06cbe0 2023-07-04T18:52:50.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.118+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c1480c0 con 0x7f404c06cbe0 2023-07-04T18:52:50.126 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.119+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c1482e0 con 0x7f404c06cbe0 2023-07-04T18:52:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.119+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4040007470 con 0x7f404c06cbe0 2023-07-04T18:52:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.120+0000 7f40497fa700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.120+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f4040089300 con 0x7f404c06cbe0 2023-07-04T18:52:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.120+0000 7f404bfff700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.127 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.120+0000 7f404bfff700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 0x7f403006d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4038000f60 tx=0x7f4038009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.128 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.124+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4040054330 con 0x7f404c06cbe0 2023-07-04T18:52:50.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.393+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f404c04e210 con 0x7f404c06cbe0 2023-07-04T18:52:50.395 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.393+0000 7f40497fa700 1 -- 172.21.15.77:0/4108571784 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1268 (secure 0 0 0) 0x7f4040057950 con 0x7f404c06cbe0 2023-07-04T18:52:50.396 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:52:50.396 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5","modified":"2023-07-04T18:50:23.019286Z","created":"2023-07-04T18:50:23.019286Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3301","nonce":0},{"type":"v1","addr":"172.21.15.82:6790","nonce":0}]},"addr":"172.21.15.82:6790/0","public_addr":"172.21.15.82:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-07-04T18:52:50.397 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 msgr2=0x7f403006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.398 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 0x7f403006d190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4038000f60 tx=0x7f4038009040 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 msgr2=0x7f404c147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f404000c750 tx=0x7f404000cb10 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 shutdown_connections 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.395+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f403006ace0 0x7f403006d190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f404c06cbe0 0x7f404c147870 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f404c158cc0 0x7f404c149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 --2- 172.21.15.77:0/4108571784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f404c0cb940 0x7f404c1f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.399 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 >> 172.21.15.77:0/4108571784 conn(0x7f404c06aad0 msgr2=0x7f404c0ae3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:50.400 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 shutdown_connections 2023-07-04T18:52:50.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.396+0000 7f4052b4d700 1 -- 172.21.15.77:0/4108571784 wait complete. 2023-07-04T18:52:50.401 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2023-07-04T18:52:50.410 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get-or-create client.admin 2023-07-04T18:52:50.552 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.551+0000 7f5f67965700 1 Processor -- start 2023-07-04T18:52:50.553 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.551+0000 7f5f67965700 1 -- start start 2023-07-04T18:52:50.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f67965700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600d33c0 0x7f5f600b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f67965700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f600af8f0 0x7f5f600b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.563 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f67965700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600b2790 con 0x7f5f600b1860 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f67965700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f600b2a70 con 0x7f5f600af8f0 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f67965700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600b2d50 con 0x7f5f600d33c0 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f600af8f0 0x7f5f600b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f600af8f0 0x7f5f600b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46190/0 (socket says 172.21.15.77:46190) 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 -- 172.21.15.77:0/3690594740 learned_addr learned my addr 172.21.15.77:0/3690594740 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:50.564 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f64f00700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600d33c0 0x7f5f600b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 -- 172.21.15.77:0/3690594740 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600d33c0 msgr2=0x7f5f600b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600d33c0 0x7f5f600b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 -- 172.21.15.77:0/3690594740 >> v1:172.21.15.82:6789/0 conn(0x7f5f600b1860 legacy=0x7f5f600d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.552+0000 7f5f65f02700 1 -- 172.21.15.77:0/3690594740 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600b3030 con 0x7f5f600af8f0 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f65f02700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600af8f0 0x7f5f600b3d70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9482ce515d65bbf0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f57fff700 1 -- 172.21.15.77:0/3690594740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5800e070 con 0x7f5f600af8f0 2023-07-04T18:52:50.565 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f57fff700 1 -- 172.21.15.77:0/3690594740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5800b910 con 0x7f5f600af8f0 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f57fff700 1 -- 172.21.15.77:0/3690594740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5800bcc0 con 0x7f5f600af8f0 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 -- 172.21.15.77:0/3690594740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600af8f0 msgr2=0x7f5f600b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600af8f0 0x7f5f600b3d70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5800b330 comp rx=0 tx=0).stop 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 -- 172.21.15.77:0/3690594740 shutdown_connections 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600af8f0 0x7f5f600b3d70 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 --2- 172.21.15.77:0/3690594740 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600d33c0 0x7f5f600b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.566 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 -- 172.21.15.77:0/3690594740 >> 172.21.15.77:0/3690594740 conn(0x7f5f60071e90 msgr2=0x7f5f6009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 -- 172.21.15.77:0/3690594740 shutdown_connections 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.553+0000 7f5f67965700 1 -- 172.21.15.77:0/3690594740 wait complete. 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.555+0000 7f5f67965700 1 Processor -- start 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.555+0000 7f5f67965700 1 -- start start 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600af8f0 0x7f5f601f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f600d33c0 0x7f5f600cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.567 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c9a70 con 0x7f5f600b1860 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600c7820 con 0x7f5f600d33c0 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601f2340 con 0x7f5f600af8f0 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46194/0 (socket says 172.21.15.77:46194) 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 -- 172.21.15.77:0/686674276 learned_addr learned my addr 172.21.15.77:0/686674276 (peer_addr_for_me v2:172.21.15.77:0/0) 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f65701700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600af8f0 0x7f5f601f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.568 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 -- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600af8f0 msgr2=0x7f5f601f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f65f02700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f600d33c0 0x7f5f600cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600af8f0 0x7f5f601f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 -- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f600d33c0 msgr2=0x7f5f600cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f600d33c0 0x7f5f600cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 -- 172.21.15.77:0/686674276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f58009b30 con 0x7f5f600b1860 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f64f00700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000e8a0 tx=0x7f5f5000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50009590 con 0x7f5f600b1860 2023-07-04T18:52:50.569 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5000eea0 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f600c81f0 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.557+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000ca50 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.558+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f600c6ef0 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.558+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6004e210 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.559+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5f5000cbb0 con 0x7f5f600b1860 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.559+0000 7f5f55ffb700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 0x7f5f4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T18:52:50.570 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.559+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f5f50089440 con 0x7f5f600b1860 2023-07-04T18:52:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.559+0000 7f5f65701700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 0x7f5f4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T18:52:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.560+0000 7f5f65701700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 0x7f5f4406d190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c007fd0 tx=0x7f5f5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-04T18:52:50.571 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.564+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5f50057f80 con 0x7f5f600b1860 2023-07-04T18:52:50.834 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.833+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.admin"} v 0) v1 -- 0x7f5f600b24d0 con 0x7f5f600b1860 2023-07-04T18:52:50.836 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.833+0000 7f5f55ffb700 1 -- 172.21.15.77:0/686674276 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.admin"}]=0 v11) v1 ==== 92+0+63 (secure 0 0 0) 0x7f5f50057b10 con 0x7f5f600b1860 2023-07-04T18:52:50.836 INFO:teuthology.orchestra.run.smithi077.stdout:[client.admin] 2023-07-04T18:52:50.836 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== 2023-07-04T18:52:50.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 msgr2=0x7f5f4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 0x7f5f4406d190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c007fd0 tx=0x7f5f5c009040 comp rx=0 tx=0).stop 2023-07-04T18:52:50.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 msgr2=0x7f5f600c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T18:52:50.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000e8a0 tx=0x7f5f5000ec60 comp rx=0 tx=0).stop 2023-07-04T18:52:50.838 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 shutdown_connections 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:6832/3872286367,v1:172.21.15.82:6833/3872286367] conn(0x7f5f4406ace0 0x7f5f4406d190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f600d33c0 0x7f5f600cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f5f600af8f0 0x7f5f601f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 --2- 172.21.15.77:0/686674276 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f600b1860 0x7f5f600c7cb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 >> 172.21.15.77:0/686674276 conn(0x7f5f60071e90 msgr2=0x7f5f600b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-04T18:52:50.839 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 shutdown_connections 2023-07-04T18:52:50.840 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T18:52:50.835+0000 7f5f67965700 1 -- 172.21.15.77:0/686674276 wait complete. 2023-07-04T18:52:50.849 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:52:50.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/tmp/tmp.pA346HXQMhxfstests-dev/local.config 2023-07-04T18:52:50.887 INFO:tasks.cephfs.xfstests_dev:local.config's contents - export FSTYP=ceph export TEST_DEV=172.21.15.77:6789:/test export TEST_DIR=/tmp/tmp.ujtNGq2JOmtest export SCRATCH_DEV=172.21.15.77:6789:/scratch export SCRATCH_MNT=/tmp/tmp.nhbIsfbtq4scratch export CEPHFS_MOUNT_OPTIONS="-o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A==" 2023-07-04T18:52:50.888 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T18:52:50.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/tmp/tmp.pA346HXQMhxfstests-dev/ceph.exclude 2023-07-04T18:52:50.955 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo make) 2023-07-04T18:52:51.039 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize -cfi 2023-07-04T18:52:51.360 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: putting auxiliary files in '.'. 2023-07-04T18:52:51.361 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file './config.guess' 2023-07-04T18:52:51.364 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file './config.sub' 2023-07-04T18:52:51.367 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file './install-sh' 2023-07-04T18:52:51.371 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file './ltmain.sh' 2023-07-04T18:52:51.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:52:51.395 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:52:51.396 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2023-07-04T18:52:51.396 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file 'm4/libtool.m4' 2023-07-04T18:52:51.396 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2023-07-04T18:52:51.400 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2023-07-04T18:52:51.404 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file 'm4/ltversion.m4' 2023-07-04T18:52:51.408 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2023-07-04T18:52:51.414 INFO:teuthology.orchestra.run.smithi077.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2023-07-04T18:52:51.415 INFO:teuthology.orchestra.run.smithi077.stdout:cp include/install-sh . 2023-07-04T18:52:51.417 INFO:teuthology.orchestra.run.smithi077.stdout:aclocal -I m4 2023-07-04T18:52:52.567 INFO:teuthology.orchestra.run.smithi077.stdout:autoheader 2023-07-04T18:52:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:autoconf 2023-07-04T18:52:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:./configure \ 2023-07-04T18:52:53.025 INFO:teuthology.orchestra.run.smithi077.stdout: --libexecdir=/usr/lib \ 2023-07-04T18:52:53.025 INFO:teuthology.orchestra.run.smithi077.stdout: --exec_prefix=/var/lib 2023-07-04T18:52:53.442 INFO:teuthology.orchestra.run.smithi077.stdout:checking build system type... x86_64-pc-linux-gnu 2023-07-04T18:52:53.442 INFO:teuthology.orchestra.run.smithi077.stdout:checking host system type... x86_64-pc-linux-gnu 2023-07-04T18:52:53.444 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to print strings... printf 2023-07-04T18:52:53.444 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gcc... gcc 2023-07-04T18:52:53.568 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether the C compiler works... yes 2023-07-04T18:52:53.569 INFO:teuthology.orchestra.run.smithi077.stdout:checking for C compiler default output file name... a.out 2023-07-04T18:52:53.595 INFO:teuthology.orchestra.run.smithi077.stdout:checking for suffix of executables... 2023-07-04T18:52:53.666 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether we are cross compiling... no 2023-07-04T18:52:53.683 INFO:teuthology.orchestra.run.smithi077.stdout:checking for suffix of object files... o 2023-07-04T18:52:53.699 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether we are using the GNU C compiler... yes 2023-07-04T18:52:53.714 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether gcc accepts -g... yes 2023-07-04T18:52:53.740 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gcc option to accept ISO C89... none needed 2023-07-04T18:52:53.743 INFO:teuthology.orchestra.run.smithi077.stdout:checking for a sed that does not truncate output... /bin/sed 2023-07-04T18:52:53.745 INFO:teuthology.orchestra.run.smithi077.stdout:checking for grep that handles long lines and -e... /bin/grep 2023-07-04T18:52:53.746 INFO:teuthology.orchestra.run.smithi077.stdout:checking for egrep... /bin/grep -E 2023-07-04T18:52:53.747 INFO:teuthology.orchestra.run.smithi077.stdout:checking for fgrep... /bin/grep -F 2023-07-04T18:52:53.750 INFO:teuthology.orchestra.run.smithi077.stdout:checking for ld used by gcc... /bin/ld 2023-07-04T18:52:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:checking if the linker (/bin/ld) is GNU ld... yes 2023-07-04T18:52:53.757 INFO:teuthology.orchestra.run.smithi077.stdout:checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B 2023-07-04T18:52:53.773 INFO:teuthology.orchestra.run.smithi077.stdout:checking the name lister (/bin/nm -B) interface... BSD nm 2023-07-04T18:52:53.774 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether ln -s works... yes 2023-07-04T18:52:53.784 INFO:teuthology.orchestra.run.smithi077.stdout:checking the maximum length of command line arguments... 1572864 2023-07-04T18:52:53.786 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2023-07-04T18:52:53.786 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2023-07-04T18:52:53.787 INFO:teuthology.orchestra.run.smithi077.stdout:checking for /bin/ld option to reload object files... -r 2023-07-04T18:52:53.788 INFO:teuthology.orchestra.run.smithi077.stdout:checking for objdump... objdump 2023-07-04T18:52:53.788 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to recognize dependent libraries... pass_all 2023-07-04T18:52:53.788 INFO:teuthology.orchestra.run.smithi077.stdout:checking for dlltool... no 2023-07-04T18:52:53.789 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to associate runtime and link libraries... printf %s\n 2023-07-04T18:52:53.789 INFO:teuthology.orchestra.run.smithi077.stdout:checking for ar... ar 2023-07-04T18:52:53.810 INFO:teuthology.orchestra.run.smithi077.stdout:checking for archiver @FILE support... @ 2023-07-04T18:52:53.810 INFO:teuthology.orchestra.run.smithi077.stdout:checking for strip... strip 2023-07-04T18:52:53.811 INFO:teuthology.orchestra.run.smithi077.stdout:checking for ranlib... ranlib 2023-07-04T18:52:53.811 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gawk... gawk 2023-07-04T18:52:53.860 INFO:teuthology.orchestra.run.smithi077.stdout:checking command to parse /bin/nm -B output from gcc object... ok 2023-07-04T18:52:53.861 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sysroot... no 2023-07-04T18:52:53.864 INFO:teuthology.orchestra.run.smithi077.stdout:checking for a working dd... /bin/dd 2023-07-04T18:52:53.866 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to truncate binary pipes... /bin/dd bs=4096 count=1 2023-07-04T18:52:53.879 INFO:teuthology.orchestra.run.smithi077.stdout:checking for mt... no 2023-07-04T18:52:53.881 INFO:teuthology.orchestra.run.smithi077.stdout:checking if : is a manifest tool... no 2023-07-04T18:52:53.911 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to run the C preprocessor... gcc -E 2023-07-04T18:52:54.053 INFO:teuthology.orchestra.run.smithi077.stdout:checking for ANSI C header files... yes 2023-07-04T18:52:54.072 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/types.h... yes 2023-07-04T18:52:54.099 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/stat.h... yes 2023-07-04T18:52:54.122 INFO:teuthology.orchestra.run.smithi077.stdout:checking for stdlib.h... yes 2023-07-04T18:52:54.145 INFO:teuthology.orchestra.run.smithi077.stdout:checking for string.h... yes 2023-07-04T18:52:54.169 INFO:teuthology.orchestra.run.smithi077.stdout:checking for memory.h... yes 2023-07-04T18:52:54.193 INFO:teuthology.orchestra.run.smithi077.stdout:checking for strings.h... yes 2023-07-04T18:52:54.219 INFO:teuthology.orchestra.run.smithi077.stdout:checking for inttypes.h... yes 2023-07-04T18:52:54.247 INFO:teuthology.orchestra.run.smithi077.stdout:checking for stdint.h... yes 2023-07-04T18:52:54.274 INFO:teuthology.orchestra.run.smithi077.stdout:checking for unistd.h... yes 2023-07-04T18:52:54.299 INFO:teuthology.orchestra.run.smithi077.stdout:checking for dlfcn.h... yes 2023-07-04T18:52:54.302 INFO:teuthology.orchestra.run.smithi077.stdout:checking for objdir... .libs 2023-07-04T18:52:54.367 INFO:teuthology.orchestra.run.smithi077.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2023-07-04T18:52:54.367 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2023-07-04T18:52:54.382 INFO:teuthology.orchestra.run.smithi077.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2023-07-04T18:52:54.400 INFO:teuthology.orchestra.run.smithi077.stdout:checking if gcc static flag -static works... no 2023-07-04T18:52:54.420 INFO:teuthology.orchestra.run.smithi077.stdout:checking if gcc supports -c -o file.o... yes 2023-07-04T18:52:54.420 INFO:teuthology.orchestra.run.smithi077.stdout:checking if gcc supports -c -o file.o... (cached) yes 2023-07-04T18:52:54.426 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes 2023-07-04T18:52:54.450 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether -lc should be explicitly linked in... no 2023-07-04T18:52:54.507 INFO:teuthology.orchestra.run.smithi077.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2023-07-04T18:52:54.508 INFO:teuthology.orchestra.run.smithi077.stdout:checking how to hardcode library paths into programs... immediate 2023-07-04T18:52:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether stripping libraries is possible... yes 2023-07-04T18:52:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:checking if libtool supports shared libraries... yes 2023-07-04T18:52:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether to build shared libraries... yes 2023-07-04T18:52:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether to build static libraries... yes 2023-07-04T18:52:54.514 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gcc... (cached) gcc 2023-07-04T18:52:54.531 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-07-04T18:52:54.531 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether gcc accepts -g... (cached) yes 2023-07-04T18:52:54.531 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-07-04T18:52:54.532 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gmake... /usr/bin/gmake 2023-07-04T18:52:54.533 INFO:teuthology.orchestra.run.smithi077.stderr:./configure: line 12070: test: $(SHELL): binary operator expected 2023-07-04T18:52:54.534 INFO:teuthology.orchestra.run.smithi077.stdout:checking for tar... /bin/tar 2023-07-04T18:52:54.534 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gzip... /bin/gzip 2023-07-04T18:52:54.537 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether gcc -MM is supported... yes 2023-07-04T18:52:54.537 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sort... /bin/sort 2023-07-04T18:52:54.537 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether ln -s works... yes 2023-07-04T18:52:54.538 INFO:teuthology.orchestra.run.smithi077.stdout:checking for rpm... /bin/rpm 2023-07-04T18:52:54.541 INFO:teuthology.orchestra.run.smithi077.stdout:checking for rpmbuild... no 2023-07-04T18:52:54.595 INFO:teuthology.orchestra.run.smithi077.stdout:checking assert.h usability... yes 2023-07-04T18:52:54.605 INFO:teuthology.orchestra.run.smithi077.stdout:checking assert.h presence... yes 2023-07-04T18:52:54.605 INFO:teuthology.orchestra.run.smithi077.stdout:checking for assert.h... yes 2023-07-04T18:52:54.628 INFO:teuthology.orchestra.run.smithi077.stdout:checking bstring.h usability... no 2023-07-04T18:52:54.637 INFO:teuthology.orchestra.run.smithi077.stdout:checking bstring.h presence... no 2023-07-04T18:52:54.637 INFO:teuthology.orchestra.run.smithi077.stdout:checking for bstring.h... no 2023-07-04T18:52:54.661 INFO:teuthology.orchestra.run.smithi077.stdout:checking libgen.h usability... yes 2023-07-04T18:52:54.667 INFO:teuthology.orchestra.run.smithi077.stdout:checking libgen.h presence... yes 2023-07-04T18:52:54.668 INFO:teuthology.orchestra.run.smithi077.stdout:checking for libgen.h... yes 2023-07-04T18:52:54.695 INFO:teuthology.orchestra.run.smithi077.stdout:checking dirent.h usability... yes 2023-07-04T18:52:54.702 INFO:teuthology.orchestra.run.smithi077.stdout:checking dirent.h presence... yes 2023-07-04T18:52:54.703 INFO:teuthology.orchestra.run.smithi077.stdout:checking for dirent.h... yes 2023-07-04T18:52:54.754 INFO:teuthology.orchestra.run.smithi077.stdout:checking errno.h usability... yes 2023-07-04T18:52:54.761 INFO:teuthology.orchestra.run.smithi077.stdout:checking errno.h presence... yes 2023-07-04T18:52:54.761 INFO:teuthology.orchestra.run.smithi077.stdout:checking for errno.h... yes 2023-07-04T18:52:54.788 INFO:teuthology.orchestra.run.smithi077.stdout:checking malloc.h usability... yes 2023-07-04T18:52:54.796 INFO:teuthology.orchestra.run.smithi077.stdout:checking malloc.h presence... yes 2023-07-04T18:52:54.796 INFO:teuthology.orchestra.run.smithi077.stdout:checking for malloc.h... yes 2023-07-04T18:52:54.825 INFO:teuthology.orchestra.run.smithi077.stdout:checking uuid.h usability... no 2023-07-04T18:52:54.831 INFO:teuthology.orchestra.run.smithi077.stdout:checking uuid.h presence... no 2023-07-04T18:52:54.831 INFO:teuthology.orchestra.run.smithi077.stdout:checking for uuid.h... no 2023-07-04T18:52:54.856 INFO:teuthology.orchestra.run.smithi077.stdout:checking uuid/uuid.h usability... yes 2023-07-04T18:52:54.865 INFO:teuthology.orchestra.run.smithi077.stdout:checking uuid/uuid.h presence... yes 2023-07-04T18:52:54.865 INFO:teuthology.orchestra.run.smithi077.stdout:checking for uuid/uuid.h... yes 2023-07-04T18:52:54.890 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/uuid.h usability... no 2023-07-04T18:52:54.899 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/uuid.h presence... no 2023-07-04T18:52:54.899 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/uuid.h... no 2023-07-04T18:52:54.925 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/file.h usability... yes 2023-07-04T18:52:54.933 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/file.h presence... yes 2023-07-04T18:52:54.933 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/file.h... yes 2023-07-04T18:52:54.958 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fcntl.h usability... yes 2023-07-04T18:52:54.965 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fcntl.h presence... yes 2023-07-04T18:52:54.966 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/fcntl.h... yes 2023-07-04T18:52:54.991 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/syssgi.h usability... no 2023-07-04T18:52:55.004 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/syssgi.h presence... no 2023-07-04T18:52:55.005 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/syssgi.h... no 2023-07-04T18:52:55.040 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/param.h usability... yes 2023-07-04T18:52:55.050 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/param.h presence... yes 2023-07-04T18:52:55.050 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/param.h... yes 2023-07-04T18:52:55.054 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/stat.h... (cached) yes 2023-07-04T18:52:55.081 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/statvfs.h usability... yes 2023-07-04T18:52:55.088 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/statvfs.h presence... yes 2023-07-04T18:52:55.088 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/statvfs.h... yes 2023-07-04T18:52:55.113 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/time.h usability... yes 2023-07-04T18:52:55.121 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/time.h presence... yes 2023-07-04T18:52:55.121 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/time.h... yes 2023-07-04T18:52:55.155 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/ioctl.h usability... yes 2023-07-04T18:52:55.162 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/ioctl.h presence... yes 2023-07-04T18:52:55.163 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/ioctl.h... yes 2023-07-04T18:52:55.191 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/wait.h usability... yes 2023-07-04T18:52:55.199 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/wait.h presence... yes 2023-07-04T18:52:55.199 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/wait.h... yes 2023-07-04T18:52:55.204 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/types.h... (cached) yes 2023-07-04T18:52:55.208 INFO:teuthology.orchestra.run.smithi077.stdout:checking for strings.h... (cached) yes 2023-07-04T18:52:55.237 INFO:teuthology.orchestra.run.smithi077.stdout:checking err.h usability... yes 2023-07-04T18:52:55.243 INFO:teuthology.orchestra.run.smithi077.stdout:checking err.h presence... yes 2023-07-04T18:52:55.244 INFO:teuthology.orchestra.run.smithi077.stdout:checking for err.h... yes 2023-07-04T18:52:55.270 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/falloc.h usability... yes 2023-07-04T18:52:55.276 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/falloc.h presence... yes 2023-07-04T18:52:55.276 INFO:teuthology.orchestra.run.smithi077.stdout:checking for linux/falloc.h... yes 2023-07-04T18:52:55.301 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fs/xfs_fsops.h usability... no 2023-07-04T18:52:55.310 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fs/xfs_fsops.h presence... no 2023-07-04T18:52:55.310 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/fs/xfs_fsops.h... no 2023-07-04T18:52:55.333 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fs/xfs_itable.h usability... no 2023-07-04T18:52:55.342 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/fs/xfs_itable.h presence... no 2023-07-04T18:52:55.342 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/fs/xfs_itable.h... no 2023-07-04T18:52:55.365 INFO:teuthology.orchestra.run.smithi077.stdout:checking xfs/platform_defs.h usability... no 2023-07-04T18:52:55.374 INFO:teuthology.orchestra.run.smithi077.stdout:checking xfs/platform_defs.h presence... no 2023-07-04T18:52:55.374 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/platform_defs.h... no 2023-07-04T18:52:55.397 INFO:teuthology.orchestra.run.smithi077.stdout:checking btrfs/ioctl.h usability... no 2023-07-04T18:52:55.406 INFO:teuthology.orchestra.run.smithi077.stdout:checking btrfs/ioctl.h presence... no 2023-07-04T18:52:55.406 INFO:teuthology.orchestra.run.smithi077.stdout:checking for btrfs/ioctl.h... no 2023-07-04T18:52:55.429 INFO:teuthology.orchestra.run.smithi077.stdout:checking cifs/ioctl.h usability... no 2023-07-04T18:52:55.441 INFO:teuthology.orchestra.run.smithi077.stdout:checking cifs/ioctl.h presence... no 2023-07-04T18:52:55.442 INFO:teuthology.orchestra.run.smithi077.stdout:checking for cifs/ioctl.h... no 2023-07-04T18:52:55.462 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/mman.h usability... yes 2023-07-04T18:52:55.470 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/mman.h presence... yes 2023-07-04T18:52:55.471 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/mman.h... yes 2023-07-04T18:52:55.485 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/xfs_log_format.h... no 2023-07-04T18:52:55.518 INFO:teuthology.orchestra.run.smithi077.stdout:checking for uuid_compare... no 2023-07-04T18:52:55.547 INFO:teuthology.orchestra.run.smithi077.stdout:checking for uuid_compare in -luuid... yes 2023-07-04T18:52:55.593 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/xfs.h... yes 2023-07-04T18:52:55.606 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/libxfs.h... no 2023-07-04T18:52:55.621 INFO:teuthology.orchestra.run.smithi077.stdout:checking whether xlog_assign_lsn is declared... no 2023-07-04T18:52:55.647 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/xqm.h... yes 2023-07-04T18:52:55.691 INFO:teuthology.orchestra.run.smithi077.stdout:checking xfsctl from xfs/xfs.h... ok 2023-07-04T18:52:55.713 INFO:teuthology.orchestra.run.smithi077.stdout:checking xfs/handle.h usability... yes 2023-07-04T18:52:55.719 INFO:teuthology.orchestra.run.smithi077.stdout:checking xfs/handle.h presence... yes 2023-07-04T18:52:55.719 INFO:teuthology.orchestra.run.smithi077.stdout:checking for xfs/handle.h... yes 2023-07-04T18:52:55.755 INFO:teuthology.orchestra.run.smithi077.stdout:checking for attr_list_by_handle in -lhandle... yes 2023-07-04T18:52:55.782 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/acl.h usability... yes 2023-07-04T18:52:55.790 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/acl.h presence... yes 2023-07-04T18:52:55.790 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/acl.h... yes 2023-07-04T18:52:55.820 INFO:teuthology.orchestra.run.smithi077.stdout:checking acl/libacl.h usability... yes 2023-07-04T18:52:55.827 INFO:teuthology.orchestra.run.smithi077.stdout:checking acl/libacl.h presence... yes 2023-07-04T18:52:55.828 INFO:teuthology.orchestra.run.smithi077.stdout:checking for acl/libacl.h... yes 2023-07-04T18:52:55.858 INFO:teuthology.orchestra.run.smithi077.stdout:checking for acl_init in -lacl... yes 2023-07-04T18:52:55.880 INFO:teuthology.orchestra.run.smithi077.stdout:checking gdbm-ndbm.h usability... no 2023-07-04T18:52:55.889 INFO:teuthology.orchestra.run.smithi077.stdout:checking gdbm-ndbm.h presence... no 2023-07-04T18:52:55.890 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gdbm-ndbm.h... no 2023-07-04T18:52:55.910 INFO:teuthology.orchestra.run.smithi077.stdout:checking gdbm/ndbm.h usability... no 2023-07-04T18:52:55.919 INFO:teuthology.orchestra.run.smithi077.stdout:checking gdbm/ndbm.h presence... no 2023-07-04T18:52:55.919 INFO:teuthology.orchestra.run.smithi077.stdout:checking for gdbm/ndbm.h... no 2023-07-04T18:52:55.939 INFO:teuthology.orchestra.run.smithi077.stdout:checking ndbm.h usability... no 2023-07-04T18:52:55.952 INFO:teuthology.orchestra.run.smithi077.stdout:checking ndbm.h presence... no 2023-07-04T18:52:55.953 INFO:teuthology.orchestra.run.smithi077.stdout:checking for ndbm.h... no 2023-07-04T18:52:55.973 INFO:teuthology.orchestra.run.smithi077.stdout:checking libaio.h usability... yes 2023-07-04T18:52:55.983 INFO:teuthology.orchestra.run.smithi077.stdout:checking libaio.h presence... yes 2023-07-04T18:52:55.984 INFO:teuthology.orchestra.run.smithi077.stdout:checking for libaio.h... yes 2023-07-04T18:52:56.005 INFO:teuthology.orchestra.run.smithi077.stdout:checking liburing.h usability... no 2023-07-04T18:52:56.014 INFO:teuthology.orchestra.run.smithi077.stdout:checking liburing.h presence... no 2023-07-04T18:52:56.014 INFO:teuthology.orchestra.run.smithi077.stdout:checking for liburing.h... no 2023-07-04T18:52:56.035 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/capability.h usability... no 2023-07-04T18:52:56.044 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/capability.h presence... no 2023-07-04T18:52:56.044 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/capability.h... no 2023-07-04T18:52:56.066 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/fiemap.h usability... yes 2023-07-04T18:52:56.072 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/fiemap.h presence... yes 2023-07-04T18:52:56.073 INFO:teuthology.orchestra.run.smithi077.stdout:checking for linux/fiemap.h... yes 2023-07-04T18:52:56.105 INFO:teuthology.orchestra.run.smithi077.stdout:checking for fallocate... yes 2023-07-04T18:52:56.137 INFO:teuthology.orchestra.run.smithi077.stdout:checking for open_by_handle_at... yes 2023-07-04T18:52:56.160 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/prctl.h usability... yes 2023-07-04T18:52:56.168 INFO:teuthology.orchestra.run.smithi077.stdout:checking sys/prctl.h presence... yes 2023-07-04T18:52:56.168 INFO:teuthology.orchestra.run.smithi077.stdout:checking for sys/prctl.h... yes 2023-07-04T18:52:56.191 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/fs.h usability... yes 2023-07-04T18:52:56.198 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/fs.h presence... yes 2023-07-04T18:52:56.198 INFO:teuthology.orchestra.run.smithi077.stdout:checking for linux/fs.h... yes 2023-07-04T18:52:56.218 INFO:teuthology.orchestra.run.smithi077.stdout:checking btrfsutil.h usability... no 2023-07-04T18:52:56.228 INFO:teuthology.orchestra.run.smithi077.stdout:checking btrfsutil.h presence... no 2023-07-04T18:52:56.228 INFO:teuthology.orchestra.run.smithi077.stdout:checking for btrfsutil.h... no 2023-07-04T18:52:56.270 INFO:teuthology.orchestra.run.smithi077.stdout:checking for copy_file_range... yes 2023-07-04T18:52:56.303 INFO:teuthology.orchestra.run.smithi077.stdout:checking for SEEK_DATA... yes 2023-07-04T18:52:56.347 INFO:teuthology.orchestra.run.smithi077.stdout:checking for BMV_OF_SHARED... yes 2023-07-04T18:52:56.385 INFO:teuthology.orchestra.run.smithi077.stdout:checking for nftw... yes 2023-07-04T18:52:56.444 INFO:teuthology.orchestra.run.smithi077.stdout:checking for RLIMIT_NOFILE... yes 2023-07-04T18:52:56.466 INFO:teuthology.orchestra.run.smithi077.stdout:checking for FIEXCHANGE_RANGE... no 2023-07-04T18:52:56.501 INFO:teuthology.orchestra.run.smithi077.stdout:checking for renameat2... yes 2023-07-04T18:52:56.529 INFO:teuthology.orchestra.run.smithi077.stdout:checking for reallocarray... yes 2023-07-04T18:52:56.542 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct mount_attr... no 2023-07-04T18:52:56.590 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_qgroup_limit... yes 2023-07-04T18:52:56.623 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_qgroup_inherit... yes 2023-07-04T18:52:56.657 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_vol_args... yes 2023-07-04T18:52:56.690 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2023-07-04T18:52:56.723 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2023-07-04T18:52:56.756 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2023-07-04T18:52:56.789 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2023-07-04T18:52:56.815 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/btrfs.h usability... yes 2023-07-04T18:52:56.822 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/btrfs.h presence... yes 2023-07-04T18:52:56.822 INFO:teuthology.orchestra.run.smithi077.stdout:checking for linux/btrfs.h... yes 2023-07-04T18:52:56.850 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/btrfs_tree.h usability... yes 2023-07-04T18:52:56.858 INFO:teuthology.orchestra.run.smithi077.stdout:checking linux/btrfs_tree.h presence... yes 2023-07-04T18:52:56.858 INFO:teuthology.orchestra.run.smithi077.stdout:checking for linux/btrfs_tree.h... yes 2023-07-04T18:52:56.888 INFO:teuthology.orchestra.run.smithi077.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... no 2023-07-04T18:52:56.904 INFO:teuthology.orchestra.run.smithi077.stdout:configure: creating ./config.status 2023-07-04T18:52:57.199 INFO:teuthology.orchestra.run.smithi077.stdout:config.status: creating include/builddefs 2023-07-04T18:52:57.209 INFO:teuthology.orchestra.run.smithi077.stdout:config.status: creating include/config.h 2023-07-04T18:52:57.217 INFO:teuthology.orchestra.run.smithi077.stdout:config.status: executing libtool commands 2023-07-04T18:52:57.253 INFO:teuthology.orchestra.run.smithi077.stdout:Building include 2023-07-04T18:52:57.257 INFO:teuthology.orchestra.run.smithi077.stdout:Building lib 2023-07-04T18:52:57.260 INFO:teuthology.orchestra.run.smithi077.stdout: [LTDEP] 2023-07-04T18:52:57.360 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dataascii.lo 2023-07-04T18:52:57.470 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] databin.lo 2023-07-04T18:52:57.608 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] datapid.lo 2023-07-04T18:52:57.701 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] file_lock.lo 2023-07-04T18:52:57.836 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] forker.lo 2023-07-04T18:52:57.960 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] pattern.lo 2023-07-04T18:52:58.072 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] open_flags.lo 2023-07-04T18:52:58.203 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] random_range.lo 2023-07-04T18:52:58.409 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] string_to_tokens.lo 2023-07-04T18:52:58.507 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] str_to_bytes.lo 2023-07-04T18:52:58.626 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] tlibio.lo 2023-07-04T18:52:58.835 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] write_log.lo 2023-07-04T18:52:59.052 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] random.lo 2023-07-04T18:52:59.157 INFO:teuthology.orchestra.run.smithi077.stdout: [LD] 2023-07-04T18:52:59.243 INFO:teuthology.orchestra.run.smithi077.stdout:Building ltp 2023-07-04T18:52:59.246 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:52:59.370 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] doio 2023-07-04T18:52:59.956 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fsstress 2023-07-04T18:53:01.132 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fsx 2023-07-04T18:53:01.907 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] growfiles 2023-07-04T18:53:02.361 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] iogen 2023-07-04T18:53:02.683 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-stress 2023-07-04T18:53:03.056 INFO:teuthology.orchestra.run.smithi077.stdout:Building src 2023-07-04T18:53:03.061 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:53:04.361 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:53:04.387 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:53:04.394 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:53:04.572 INFO:teuthology.orchestra.run.smithi077.stdout: [DEP] 2023-07-04T18:53:04.593 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dirstress 2023-07-04T18:53:04.804 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fill 2023-07-04T18:53:04.943 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fill2 2023-07-04T18:53:05.147 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] getpagesize 2023-07-04T18:53:05.278 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] holes 2023-07-04T18:53:05.452 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] lstat64 2023-07-04T18:53:05.620 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] nametest 2023-07-04T18:53:05.833 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] permname 2023-07-04T18:53:06.010 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] randholes 2023-07-04T18:53:06.232 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] runas 2023-07-04T18:53:06.389 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] truncfile 2023-07-04T18:53:06.550 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] usemem 2023-07-04T18:53:06.687 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] mmapcat 2023-07-04T18:53:06.822 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] append_reader 2023-07-04T18:53:06.958 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] append_writer 2023-07-04T18:53:07.095 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dirperf 2023-07-04T18:53:07.293 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] metaperf 2023-07-04T18:53:07.523 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] devzero 2023-07-04T18:53:07.680 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] feature 2023-07-04T18:53:07.872 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] alloc 2023-07-04T18:53:08.061 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fault 2023-07-04T18:53:08.212 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fstest 2023-07-04T18:53:08.422 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_access_root 2023-07-04T18:53:08.563 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] godown 2023-07-04T18:53:08.716 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] resvtest 2023-07-04T18:53:08.882 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] writemod 2023-07-04T18:53:09.014 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] writev_on_pagefault 2023-07-04T18:53:09.157 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] makeextents 2023-07-04T18:53:09.317 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] itrash 2023-07-04T18:53:09.459 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] rename 2023-07-04T18:53:09.589 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] multi_open_unlink 2023-07-04T18:53:09.737 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] unwritten_sync 2023-07-04T18:53:09.898 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] genhashnames 2023-07-04T18:53:10.068 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_holes 2023-07-04T18:53:10.202 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_writev 2023-07-04T18:53:10.336 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_truncate_cmtime 2023-07-04T18:53:10.481 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dirhash_collide 2023-07-04T18:53:10.645 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_rename_overwrite 2023-07-04T18:53:10.774 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] holetest 2023-07-04T18:53:10.970 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_truncate_self 2023-07-04T18:53:11.099 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] af_unix 2023-07-04T18:53:11.238 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_stale_pmd 2023-07-04T18:53:11.376 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_cow_race 2023-07-04T18:53:11.528 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_fallocate 2023-07-04T18:53:11.667 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fsync-err 2023-07-04T18:53:11.832 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_write_ro 2023-07-04T18:53:11.970 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_ext4_dax_journal_corruption 2023-07-04T18:53:12.115 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_ext4_dax_inline_corruption 2023-07-04T18:53:12.256 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_ofd_locks 2023-07-04T18:53:12.437 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_collision 2023-07-04T18:53:12.615 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] mmap-write-concurrent 2023-07-04T18:53:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_get_file_time 2023-07-04T18:53:12.902 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_create_short_dirs 2023-07-04T18:53:13.054 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_create_long_dirs 2023-07-04T18:53:13.203 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_enospc 2023-07-04T18:53:13.371 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_writev_overlap 2023-07-04T18:53:13.525 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] checkpoint_journal 2023-07-04T18:53:13.666 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] mmap-rw-fault 2023-07-04T18:53:13.827 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] allocstale 2023-07-04T18:53:13.979 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_cow_memory_failure 2023-07-04T18:53:14.133 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fake-dump-rootino 2023-07-04T18:53:14.269 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dio-buf-fault 2023-07-04T18:53:14.411 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] xfsctl 2023-07-04T18:53:14.580 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] bstat 2023-07-04T18:53:14.763 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mtab 2023-07-04T18:53:14.928 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] getdevicesize 2023-07-04T18:53:15.059 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] preallo_rw_pattern_reader 2023-07-04T18:53:15.203 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] preallo_rw_pattern_writer 2023-07-04T18:53:15.347 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] ftrunc 2023-07-04T18:53:15.483 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] trunc 2023-07-04T18:53:15.629 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fs_perms 2023-07-04T18:53:15.778 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] testx 2023-07-04T18:53:15.900 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] looptest 2023-07-04T18:53:16.074 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] locktest 2023-07-04T18:53:16.349 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] unwritten_mmap 2023-07-04T18:53:16.496 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] bulkstat_unlink_test 2023-07-04T18:53:16.678 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] deduperace 2023-07-04T18:53:16.852 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] bulkstat_unlink_test_modified 2023-07-04T18:53:17.030 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_dir_offset 2023-07-04T18:53:17.164 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_futimens 2023-07-04T18:53:17.293 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_immutable 2023-07-04T18:53:17.833 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] stale_handle 2023-07-04T18:53:17.996 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] pwrite_mmap_blocked 2023-07-04T18:53:18.134 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_dir_offset2 2023-07-04T18:53:18.307 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] seek_sanity_test 2023-07-04T18:53:18.679 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] seek_copy_test 2023-07-04T18:53:18.838 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_readdir_1 2023-07-04T18:53:18.974 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_readdir_2 2023-07-04T18:53:19.111 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fsync-tester 2023-07-04T18:53:19.335 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] nsexec 2023-07-04T18:53:19.494 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] cloner 2023-07-04T18:53:19.656 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] renameat2 2023-07-04T18:53:19.806 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_getcwd 2023-07-04T18:53:19.950 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] e4compact 2023-07-04T18:53:20.131 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] test-nextquota 2023-07-04T18:53:20.284 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] punch-alternating 2023-07-04T18:53:20.438 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] attr-list-by-handle-cursor-test 2023-07-04T18:53:20.656 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] listxattr 2023-07-04T18:53:20.903 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dio-interleaved 2023-07-04T18:53:21.052 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_dir_type 2023-07-04T18:53:21.191 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] dio-invalidate-cache 2023-07-04T18:53:21.374 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] stat_test 2023-07-04T18:53:21.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:53:21.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:53:21.644 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_encrypted_d_revalidate 2023-07-04T18:53:21.794 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] attr_replace_test 2023-07-04T18:53:21.944 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] swapon 2023-07-04T18:53:22.092 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] mkswap 2023-07-04T18:53:22.231 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_attr_corruption 2023-07-04T18:53:22.367 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_open_tmpfiles 2023-07-04T18:53:22.529 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fscrypt-crypt-util 2023-07-04T18:53:23.095 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] bulkstat_null_ocount 2023-07-04T18:53:23.239 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] splice-test 2023-07-04T18:53:23.404 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] chprojid_fail 2023-07-04T18:53:23.552 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] detached_mounts_propagation 2023-07-04T18:53:23.722 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] ext4_resize 2023-07-04T18:53:23.857 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_readdir_3 2023-07-04T18:53:24.034 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] splice2pipe 2023-07-04T18:53:24.182 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] uuid_ioctl 2023-07-04T18:53:24.322 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fiemap-tester 2023-07-04T18:53:24.565 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_stripealign 2023-07-04T18:53:24.706 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] open_by_handle 2023-07-04T18:53:24.935 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] t_mmap_dio 2023-07-04T18:53:25.082 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] xfsfind 2023-07-04T18:53:25.257 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] fssum 2023-07-04T18:53:25.643 INFO:teuthology.orchestra.run.smithi077.stdout:Building log-writes 2023-07-04T18:53:25.648 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] replay-log 2023-07-04T18:53:25.927 INFO:teuthology.orchestra.run.smithi077.stdout:Building perf 2023-07-04T18:53:25.930 INFO:teuthology.orchestra.run.smithi077.stdout:Building aio-dio-regress 2023-07-04T18:53:25.936 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-append-write-fallocate-race 2023-07-04T18:53:26.094 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-fcntl-race 2023-07-04T18:53:26.245 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-subblock-eof-read 2023-07-04T18:53:26.380 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-append-write-read-race 2023-07-04T18:53:26.548 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-invalidate-readahead 2023-07-04T18:53:26.700 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-write-verify 2023-07-04T18:53:26.876 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-free-ring-with-bogus-nr-pages 2023-07-04T18:53:27.002 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-last-ref-held-by-io 2023-07-04T18:53:27.163 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-cycle-write 2023-07-04T18:53:27.334 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-extend-stat 2023-07-04T18:53:27.486 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-io-setup-with-nonwritable-context-pointer 2023-07-04T18:53:27.611 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-hole-filling-race 2023-07-04T18:53:27.750 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-invalidate-failure 2023-07-04T18:53:27.898 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aiodio_sparse2 2023-07-04T18:53:28.096 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aiocp 2023-07-04T18:53:28.301 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-cow-race 2023-07-04T18:53:28.443 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] aio-dio-eof-race 2023-07-04T18:53:28.613 INFO:teuthology.orchestra.run.smithi077.stdout:Building vfs 2023-07-04T18:53:28.618 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] vfstest 2023-07-04T18:53:30.273 INFO:teuthology.orchestra.run.smithi077.stdout: [CC] mount-idmapped 2023-07-04T18:53:30.602 INFO:teuthology.orchestra.run.smithi077.stdout:Building m4 2023-07-04T18:53:30.606 INFO:teuthology.orchestra.run.smithi077.stdout:Building common 2023-07-04T18:53:30.609 INFO:teuthology.orchestra.run.smithi077.stdout:Building tests 2023-07-04T18:53:30.614 INFO:teuthology.orchestra.run.smithi077.stdout:Building perf 2023-07-04T18:53:30.623 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/perf/group.list 2023-07-04T18:53:30.629 INFO:teuthology.orchestra.run.smithi077.stdout:Building ext4 2023-07-04T18:53:30.635 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ext4/group.list 2023-07-04T18:53:30.668 INFO:teuthology.orchestra.run.smithi077.stdout:Building ocfs2 2023-07-04T18:53:30.672 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ocfs2/group.list 2023-07-04T18:53:30.684 INFO:teuthology.orchestra.run.smithi077.stdout:Building overlay 2023-07-04T18:53:30.690 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/overlay/group.list 2023-07-04T18:53:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:Building cifs 2023-07-04T18:53:30.727 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/cifs/group.list 2023-07-04T18:53:30.738 INFO:teuthology.orchestra.run.smithi077.stdout:Building generic 2023-07-04T18:53:30.744 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/generic/group.list 2023-07-04T18:53:30.831 INFO:teuthology.orchestra.run.smithi077.stdout:Building shared 2023-07-04T18:53:30.836 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/shared/group.list 2023-07-04T18:53:30.851 INFO:teuthology.orchestra.run.smithi077.stdout:Building nfs 2023-07-04T18:53:30.858 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/nfs/group.list 2023-07-04T18:53:30.869 INFO:teuthology.orchestra.run.smithi077.stdout:Building f2fs 2023-07-04T18:53:30.873 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/f2fs/group.list 2023-07-04T18:53:30.888 INFO:teuthology.orchestra.run.smithi077.stdout:Building udf 2023-07-04T18:53:30.893 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/udf/group.list 2023-07-04T18:53:30.903 INFO:teuthology.orchestra.run.smithi077.stdout:Building ceph 2023-07-04T18:53:30.909 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ceph/group.list 2023-07-04T18:53:30.924 INFO:teuthology.orchestra.run.smithi077.stdout:Building xfs 2023-07-04T18:53:30.926 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/xfs/group.list 2023-07-04T18:53:31.014 INFO:teuthology.orchestra.run.smithi077.stdout:Building btrfs 2023-07-04T18:53:31.020 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/btrfs/group.list 2023-07-04T18:53:31.072 INFO:teuthology.orchestra.run.smithi077.stdout:Building tmpfs 2023-07-04T18:53:31.076 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/tmpfs/group.list 2023-07-04T18:53:31.088 INFO:teuthology.orchestra.run.smithi077.stdout:Building selftest 2023-07-04T18:53:31.093 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/selftest/group.list 2023-07-04T18:53:31.105 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo make install) 2023-07-04T18:53:31.143 INFO:teuthology.orchestra.run.smithi077.stdout:Building include 2023-07-04T18:53:31.146 INFO:teuthology.orchestra.run.smithi077.stdout:Building lib 2023-07-04T18:53:31.150 INFO:teuthology.orchestra.run.smithi077.stdout:Building ltp 2023-07-04T18:53:31.153 INFO:teuthology.orchestra.run.smithi077.stdout:Building src 2023-07-04T18:53:31.185 INFO:teuthology.orchestra.run.smithi077.stdout:Building log-writes 2023-07-04T18:53:31.189 INFO:teuthology.orchestra.run.smithi077.stdout:Building perf 2023-07-04T18:53:31.192 INFO:teuthology.orchestra.run.smithi077.stdout:Building aio-dio-regress 2023-07-04T18:53:31.196 INFO:teuthology.orchestra.run.smithi077.stdout:Building vfs 2023-07-04T18:53:31.199 INFO:teuthology.orchestra.run.smithi077.stdout:Building m4 2023-07-04T18:53:31.202 INFO:teuthology.orchestra.run.smithi077.stdout:Building common 2023-07-04T18:53:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:Building tests 2023-07-04T18:53:31.211 INFO:teuthology.orchestra.run.smithi077.stdout:Building perf 2023-07-04T18:53:31.217 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/perf/group.list 2023-07-04T18:53:31.226 INFO:teuthology.orchestra.run.smithi077.stdout:Building ext4 2023-07-04T18:53:31.231 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ext4/group.list 2023-07-04T18:53:31.263 INFO:teuthology.orchestra.run.smithi077.stdout:Building ocfs2 2023-07-04T18:53:31.268 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ocfs2/group.list 2023-07-04T18:53:31.281 INFO:teuthology.orchestra.run.smithi077.stdout:Building overlay 2023-07-04T18:53:31.285 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/overlay/group.list 2023-07-04T18:53:31.319 INFO:teuthology.orchestra.run.smithi077.stdout:Building cifs 2023-07-04T18:53:31.323 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/cifs/group.list 2023-07-04T18:53:31.337 INFO:teuthology.orchestra.run.smithi077.stdout:Building generic 2023-07-04T18:53:31.344 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/generic/group.list 2023-07-04T18:53:31.429 INFO:teuthology.orchestra.run.smithi077.stdout:Building shared 2023-07-04T18:53:31.434 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/shared/group.list 2023-07-04T18:53:31.450 INFO:teuthology.orchestra.run.smithi077.stdout:Building nfs 2023-07-04T18:53:31.454 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/nfs/group.list 2023-07-04T18:53:31.467 INFO:teuthology.orchestra.run.smithi077.stdout:Building f2fs 2023-07-04T18:53:31.471 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/f2fs/group.list 2023-07-04T18:53:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:Building udf 2023-07-04T18:53:31.491 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/udf/group.list 2023-07-04T18:53:31.502 INFO:teuthology.orchestra.run.smithi077.stdout:Building ceph 2023-07-04T18:53:31.506 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/ceph/group.list 2023-07-04T18:53:31.519 INFO:teuthology.orchestra.run.smithi077.stdout:Building xfs 2023-07-04T18:53:31.525 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/xfs/group.list 2023-07-04T18:53:31.611 INFO:teuthology.orchestra.run.smithi077.stdout:Building btrfs 2023-07-04T18:53:31.617 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/btrfs/group.list 2023-07-04T18:53:31.668 INFO:teuthology.orchestra.run.smithi077.stdout:Building tmpfs 2023-07-04T18:53:31.673 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/tmpfs/group.list 2023-07-04T18:53:31.684 INFO:teuthology.orchestra.run.smithi077.stdout:Building selftest 2023-07-04T18:53:31.689 INFO:teuthology.orchestra.run.smithi077.stdout: [GROUP] /tmp/tmp.pA346HXQMhxfstests-dev/tests/selftest/group.list 2023-07-04T18:53:31.700 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C include install 2023-07-04T18:53:31.702 INFO:teuthology.orchestra.run.smithi077.stdout:gmake[1]: Nothing to be done for 'install'. 2023-07-04T18:53:31.702 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C lib install 2023-07-04T18:53:31.704 INFO:teuthology.orchestra.run.smithi077.stdout:gmake[1]: Nothing to be done for 'install'. 2023-07-04T18:53:31.704 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C ltp install 2023-07-04T18:53:31.706 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/ltp 2023-07-04T18:53:31.714 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/sh ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 doio fsstress fsx growfiles iogen aio-stress /var/lib/xfstests/ltp 2023-07-04T18:53:31.789 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 755 rwtest.sh /var/lib/xfstests/ltp 2023-07-04T18:53:31.800 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C src install 2023-07-04T18:53:31.813 INFO:teuthology.orchestra.run.smithi077.stdout:Building log-writes 2023-07-04T18:53:31.817 INFO:teuthology.orchestra.run.smithi077.stdout:Building perf 2023-07-04T18:53:31.820 INFO:teuthology.orchestra.run.smithi077.stdout:Building aio-dio-regress 2023-07-04T18:53:31.824 INFO:teuthology.orchestra.run.smithi077.stdout:Building vfs 2023-07-04T18:53:31.827 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake -C log-writes install 2023-07-04T18:53:31.828 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/log-writes 2023-07-04T18:53:31.838 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 replay-log /var/lib/xfstests/src/log-writes 2023-07-04T18:53:31.845 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake -C perf install 2023-07-04T18:53:31.846 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/perf 2023-07-04T18:53:31.856 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 ResultData.py fio-insert-and-compare.py FioResultDecoder.py FioCompare.py generate-schema.py fio-results.sql /var/lib/xfstests/src/perf 2023-07-04T18:53:31.877 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake -C aio-dio-regress install 2023-07-04T18:53:31.880 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/aio-dio-regress 2023-07-04T18:53:31.887 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 aio-dio-append-write-fallocate-race aio-dio-fcntl-race aio-dio-subblock-eof-read aio-dio-append-write-read-race aio-dio-invalidate-readahead aio-dio-write-verify aio-free-ring-with-bogus-nr-pages aio-last-ref-held-by-io aio-dio-cycle-write aio-dio-extend-stat aio-io-setup-with-nonwritable-context-pointer aio-dio-hole-filling-race aio-dio-invalidate-failure aiodio_sparse2 aiocp aio-dio-cow-race aio-dio-eof-race /var/lib/xfstests/src/aio-dio-regress 2023-07-04T18:53:31.939 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake -C vfs install 2023-07-04T18:53:31.941 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/vfs 2023-07-04T18:53:31.950 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 vfstest mount-idmapped /var/lib/xfstests/src/vfs 2023-07-04T18:53:31.962 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src 2023-07-04T18:53:31.969 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/sh ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 dirstress fill fill2 getpagesize holes lstat64 nametest permname randholes runas truncfile usemem mmapcat append_reader append_writer dirperf metaperf devzero feature alloc fault fstest t_access_root godown resvtest writemod writev_on_pagefault makeextents itrash rename multi_open_unlink unwritten_sync genhashnames t_holes t_mmap_writev t_truncate_cmtime dirhash_collide t_rename_overwrite holetest t_truncate_self af_unix t_mmap_stale_pmd t_mmap_cow_race t_mmap_fallocate fsync-err t_mmap_write_ro t_ext4_dax_journal_corruption t_ext4_dax_inline_corruption t_ofd_locks t_mmap_collision mmap-write-concurrent t_get_file_time t_create_short_dirs t_create_long_dirs t_enospc t_mmap_writev_overlap checkpoint_journal mmap-rw-fault allocstale t_mmap_cow_memory_failure fake-dump-rootino dio-buf-fault xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader preallo_rw_pattern_writer ftrunc trunc fs_perms testx looptest locktest unwritten_mmap bulkstat_unlink_test deduperace bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec cloner renameat2 t_getcwd e4compact test-nextquota punch-alternating attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type dio-invalidate-cache stat_test t_encrypted_d_revalidate attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe uuid_ioctl fiemap-tester t_stripealign open_by_handle t_mmap_dio xfsfind fssum /var/lib/xfstests/src 2023-07-04T18:53:33.144 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/sh ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 dmerror fill2attr fill2fs fill2fs_check scaleread.sh btrfs_crc32c_forged_name.py popdir.pl popattr.py /var/lib/xfstests/src 2023-07-04T18:53:33.237 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/sh ../libtool --quiet --mode=install ../install-sh -o root -g root -m 644 dumpfile /var/lib/xfstests/src 2023-07-04T18:53:33.267 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C m4 install 2023-07-04T18:53:33.269 INFO:teuthology.orchestra.run.smithi077.stdout:gmake[1]: Nothing to be done for 'install'. 2023-07-04T18:53:33.269 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C common install 2023-07-04T18:53:33.270 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/common 2023-07-04T18:53:33.279 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 644 * /var/lib/xfstests/common 2023-07-04T18:53:33.409 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C tests install 2023-07-04T18:53:33.410 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/btrfs/ install 2023-07-04T18:53:33.413 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/btrfs 2023-07-04T18:53:33.422 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 290 291 299 /var/lib/xfstests/tests/btrfs 2023-07-04T18:53:34.203 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/btrfs 2023-07-04T18:53:34.212 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 057.out 175.out 186.out 240.out 069.out 092.out 077.out 140.out 265.out 236.out 108.out 204.out 177.out 194.out 234.out 249.out 109.out 033.out 082.out 252.out 270.out 040.out 176.out 202.out 137.out 034.out 125.out 014.out 165.out 181.out 152.out 142.out 264.out 192.out 148.out 282.out 267.out 052.out 228.out 183.out 094.out 273.out 251.out 110.out 043.out 209.out 151.out 206.out 195.out 274.out 215.out 115.out 015.out 250.out 191.out 281.out 046.out 051.out 244.out 219.out 119.out 019.out 128.out 232.out 248.out 162.out 147.out 197.out 196.out 242.out 004.out 189.out 220.out 120.out 020.out 102.out 263.out 171.out 261.out 045.out 217.out 117.out 017.out 072.out 068.out 048.out 062.out 038.out 084.out 129.out 233.out 066.out 029.out 229.out 103.out 254.out 111.out 011.out 243.out 275.out 134.out 170.out 260.out 157.out 143.out 133.out 286.out 161.out 246.out 216.out 116.out 016.out 155.out 118.out 287.out 067.out 010.out 238.out 138.out 095.out 184.out 160.out 190.out 291.out 081.out 024.out 071.out 061.out 047.out 200.out 100.out 025.out 139.out 258.out 185.out 269.out 208.out 039.out 222.out 035.out 169.out 098.out 290.out 080.out 070.out 060.out 207.out 106.out 009.out 167.out 003.out 211.out 158.out 085.out 146.out 188.out 104.out 089.out 086.out 279.out 225.out 174.out 156.out 135.out 257.out 124.out 172.out 058.out 262.out 036.out 042.out 193.out 097.out 271.out 283.out 083.out 199.out 099.out 278.out 053.out 164.out 239.out 241.out 002.out 178.out 141.out 214.out 114.out 074.out 056.out 012.out 041.out 223.out 150.out 123.out 179.out 144.out 032.out 277.out 205.out 136.out 005.out 096.out 198.out 008.out 280.out 050.out 105.out 027.out 187.out 091.out 153.out 088.out 285.out 044.out 201.out 101.out 001.out 266.out 237.out 065.out 231.out 227.out 075.out 131.out 213.out 113.out 013.out 031.out 182.out 272.out 078.out 173.out 259.out 210.out 132.out 037.out 093.out 235.out 127.out 230.out 166.out 130.out 023.out 073.out 030.out 203.out 063.out 055.out 245.out 145.out 159.out 122.out 076.out 022.out 006.out 284.out 299.out 224.out 154.out 276.out 149.out 064.out 218.out 018.out 168.out 007.out 079.out 054.out 059.out 049.out 226.out 163.out 126.out 121.out 026.out 180.out 268.out 253.out 255.out 090.out 221.out 087.out 212.out 112.out 256.out 021.out 247.out 028.out 107.out /var/lib/xfstests/tests/btrfs 2023-07-04T18:53:34.996 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/ceph/ install 2023-07-04T18:53:34.997 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ceph 2023-07-04T18:53:35.006 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 /var/lib/xfstests/tests/ceph 2023-07-04T18:53:35.024 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ceph 2023-07-04T18:53:35.033 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 003.out 005.out 001.out 004.out 002.out /var/lib/xfstests/tests/ceph 2023-07-04T18:53:35.052 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/cifs/ install 2023-07-04T18:53:35.054 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/cifs 2023-07-04T18:53:35.062 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/cifs 2023-07-04T18:53:35.072 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/cifs 2023-07-04T18:53:35.079 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/cifs 2023-07-04T18:53:35.088 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/ext4/ install 2023-07-04T18:53:35.089 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ext4 2023-07-04T18:53:35.098 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 271 301 302 303 304 305 306 307 308 /var/lib/xfstests/tests/ext4 2023-07-04T18:53:35.283 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ext4 2023-07-04T18:53:35.291 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out.nodelalloc 001.out.default 001.out.nozero 001.cfg 032.out 039.out 305.out 005.out 008.out 035.out 036.out 308.out 050.out 040.out 306.out 007.out 021.out 046.out 012.out 009.out 033.out 045.out 034.out 056.out 047.out 018.out 023.out 302.out 020.out 002.out 014.out 024.out 013.out 031.out 015.out 304.out 004.out 048.out 038.out 052.out 303.out 029.out 011.out 027.out 042.out 044.out 055.out 030.out 058.out 003.out 017.out 043.out 053.out 016.out 059.out 037.out 301.out 010.out 022.out 006.out 057.out 025.out 026.out 271.out 051.out 041.out 307.out 019.out 028.out 054.out 049.out /var/lib/xfstests/tests/ext4 2023-07-04T18:53:35.489 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/f2fs/ install 2023-07-04T18:53:35.490 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/f2fs 2023-07-04T18:53:35.496 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 /var/lib/xfstests/tests/f2fs 2023-07-04T18:53:35.507 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/f2fs 2023-07-04T18:53:35.516 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out 002.out /var/lib/xfstests/tests/f2fs 2023-07-04T18:53:35.527 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/generic/ install 2023-07-04T18:53:35.531 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/generic 2023-07-04T18:53:35.540 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 /var/lib/xfstests/tests/generic 2023-07-04T18:53:37.489 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/generic 2023-07-04T18:53:37.500 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 050.out.xfsquota 035.out.nfs 035.out.default 050.out.nojournal 050.out.default 050.cfg 035.cfg 208.out 433.out 670.out 040.out 104.out 575.out 537.out 683.out 497.out 337.out 702.out 460.out 149.out 088.out 366.out 202.out 137.out 034.out 110.out 389.out 403.out 593.out 571.out 243.out 592.out 282.out 472.out 468.out 628.out 528.out 607.out 428.out 659.out 407.out 228.out 128.out 171.out 028.out 620.out 273.out 474.out 492.out 609.out 707.out 693.out 595.out 506.out 268.out 632.out 532.out 591.out 281.out 471.out 619.out 519.out 013.out 419.out 319.out 177.out 523.out 119.out 019.out 727.out 372.out 686.out 312.out 718.out 143.out 716.out 161.out 516.out 518.out 538.out 655.out 216.out 116.out 016.out 155.out 338.out 238.out 500.out 138.out 189.out 263.out 277.out 181.out 160.out 483.out 274.out 722.out 061.out 239.out 457.out 333.out 258.out 362.out 379.out 691.out 039.out 173.out 498.out 665.out 098.out 711.out 290.out 480.out 706.out 060.out 406.out 306.out 106.out 642.out 447.out 217.out 120.out 191.out 411.out 357.out 481.out 211.out 158.out 014.out 182.out 370.out 188.out 030.out 099.out 585.out 724.out 721.out 624.out 524.out 174.out 156.out 324.out 224.out 313.out 355.out 124.out 382.out 672.out 278.out 063.out 058.out 384.out 296.out 042.out 193.out 271.out 465.out 473.out 335.out 247.out 509.out 568.out 441.out 440.out 475.out 548.out 356.out 341.out 094.out 241.out 600.out 089.out 714.out 614.out 514.out 414.out 314.out 416.out 214.out 381.out 056.out 041.out 057.out 340.out 358.out 572.out 586.out 521.out 240.out 549.out 095.out 633.out 434.out 140.out 708.out 533.out 508.out 408.out 308.out 476.out 087.out 199.out 458.out 233.out 639.out 443.out 540.out 397.out 053.out 279.out 276.out 448.out 164.out 376.out 589.out 587.out 339.out 294.out 352.out 484.out 489.out 139.out 317.out 399.out 220.out 552.out 695.out 017.out 677.out 097.out 175.out 315.out 295.out 580.out 075.out 186.out 452.out 679.out 292.out 252.out 077.out 167.out 688.out 206.out 348.out 394.out 651.out 184.out 466.out 697.out 495.out 725.out 109.out 482.out 625.out 283.out 525.out 488.out 496.out 325.out 551.out 225.out 322.out 636.out 125.out 715.out 192.out 544.out 515.out 650.out 152.out 658.out 563.out 393.out 578.out 015.out 392.out 682.out 684.out 550.out 052.out 652.out 534.out 351.out 583.out 451.out 450.out 463.out 251.out 267.out 176.out 069.out 644.out 065.out 487.out 615.out 395.out 676.out 415.out 344.out 215.out 115.out 444.out 027.out 250.out 391.out 674.out 464.out 705.out 536.out 577.out 439.out 244.out 336.out 144.out 505.out 405.out 342.out 205.out 136.out 598.out 363.out 036.out 198.out 723.out 008.out 390.out 680.out 605.out 305.out 105.out 005.out 367.out 291.out 719.out 323.out 685.out 562.out 570.out 388.out 527.out 237.out 151.out 227.out 163.out 332.out 661.out 582.out 232.out 478.out 584.out 462.out 597.out 412.out 132.out 179.out 037.out 293.out 499.out 638.out 353.out 020.out 660.out 561.out 141.out 076.out 668.out 044.out 648.out 387.out 594.out 438.out 265.out 289.out 574.out 006.out 690.out 284.out 567.out 299.out 456.out 461.out 477.out 307.out 647.out 493.out 107.out 007.out 380.out 526.out 479.out 426.out 150.out 326.out 082.out 226.out 413.out 126.out 025.out 121.out 026.out 021.out 485.out 694.out 347.out 275.out 090.out 361.out 564.out 302.out 432.out 002.out 558.out 162.out 147.out 288.out 197.out 033.out 396.out 051.out 350.out 547.out 047.out 024.out 360.out 543.out 261.out 045.out 692.out 255.out 726.out 378.out 346.out 048.out 062.out 626.out 529.out 231.out 343.out 662.out 425.out 329.out 165.out 334.out 103.out 003.out 129.out 423.out 371.out 234.out 183.out 671.out 645.out 673.out 190.out 486.out 223.out 123.out 262.out 669.out 023.out 459.out 616.out 032.out 266.out 146.out 398.out 728.out 696.out 427.out 590.out 280.out 470.out 455.out 187.out 091.out 046.out 374.out 631.out 328.out 260.out 654.out 359.out 531.out 649.out 301.out 101.out 635.out 623.out 385.out 331.out 556.out 713.out 613.out 513.out 622.out 131.out 213.out 634.out 031.out 565.out 630.out 675.out 272.out 530.out 510.out 627.out 573.out 576.out 430.out 310.out 569.out 454.out 259.out 210.out 449.out 327.out 330.out 067.out 093.out 604.out 287.out 127.out 699.out 166.out 130.out 219.out 073.out 603.out 503.out 303.out 203.out 354.out 157.out 386.out 349.out 055.out 245.out 709.out 522.out 422.out 222.out 159.out 539.out 249.out 122.out 689.out 022.out 681.out 424.out 321.out 066.out 490.out 435.out 375.out 134.out 154.out 566.out 618.out 501.out 418.out 318.out 542.out 218.out 118.out 018.out 377.out 172.out 168.out 079.out 469.out 054.out 059.out 049.out 442.out 657.out 621.out 180.out 270.out 253.out 421.out 712.out 612.out 667.out 512.out 494.out 221.out 212.out 112.out 256.out 012.out 029.out 195.out 554.out 148.out 185.out 596.out 720.out 248.out 196.out 520.out 242.out 420.out 285.out 320.out 629.out 230.out 602.out 502.out 402.out 102.out 717.out 617.out 517.out 429.out 417.out 365.out 092.out 117.out 664.out 072.out 068.out 229.out 038.out 084.out 369.out 611.out 511.out 311.out 368.out 637.out 111.out 011.out 257.out 194.out 170.out 606.out 133.out 286.out 043.out 437.out 545.out 559.out 710.out 610.out 687.out 410.out 701.out 601.out 401.out 201.out 010.out 001.out 096.out 114.out 113.out 074.out 581.out 145.out 641.out 491.out 081.out 071.out 083.out 653.out 700.out 064.out 400.out 300.out 200.out 100.out 646.out 541.out 445.out 269.out 546.out 656.out 698.out 553.out 178.out 298.out 446.out 080.out 070.out 608.out 507.out 409.out 309.out 209.out 108.out 009.out 643.out 004.out 557.out 453.out 666.out 236.out 078.out 364.out 555.out 383.out 085.out 704.out 588.out 504.out 404.out 304.out 204.out 535.out 345.out 246.out 467.out 086.out 254.out 207.out 142.out 235.out 436.out 703.out 135.out 264.out 431.out 678.out 373.out 316.out 169.out 640.out 579.out 297.out 153.out 560.out 663.out 599.out /var/lib/xfstests/tests/generic 2023-07-04T18:53:39.481 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/nfs/ install 2023-07-04T18:53:39.482 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/nfs 2023-07-04T18:53:39.491 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/nfs 2023-07-04T18:53:39.499 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/nfs 2023-07-04T18:53:39.508 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/nfs 2023-07-04T18:53:39.516 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/ocfs2/ install 2023-07-04T18:53:39.518 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ocfs2 2023-07-04T18:53:39.526 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/ocfs2 2023-07-04T18:53:39.535 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ocfs2 2023-07-04T18:53:39.544 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/ocfs2 2023-07-04T18:53:39.552 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/overlay/ install 2023-07-04T18:53:39.554 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/overlay 2023-07-04T18:53:39.563 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 /var/lib/xfstests/tests/overlay 2023-07-04T18:53:39.824 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/overlay 2023-07-04T18:53:39.833 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 019.out 034.out 018.out 024.out 077.out 069.out 020.out 102.out 014.out 013.out 031.out 015.out 117.out 017.out 072.out 068.out 063.out 048.out 062.out 038.out 026.out 042.out 052.out 110.out 067.out 029.out 111.out 055.out 112.out 073.out 078.out 030.out 058.out 003.out 045.out 023.out 043.out 053.out 109.out 036.out 116.out 016.out 065.out 010.out 001.out 022.out 106.out 007.out 006.out 115.out 057.out 113.out 114.out 074.out 046.out 071.out 061.out 025.out 051.out 011.out 041.out 047.out 100.out 064.out 107.out 104.out 028.out 076.out 054.out 059.out 008.out 049.out 032.out 039.out 021.out 005.out 002.out 035.out 108.out 070.out 060.out 050.out 027.out 105.out 009.out 004.out 037.out 044.out 012.out 101.out 103.out 033.out 066.out 040.out 056.out 075.out /var/lib/xfstests/tests/overlay 2023-07-04T18:53:40.096 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/perf/ install 2023-07-04T18:53:40.098 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/perf 2023-07-04T18:53:40.107 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/perf 2023-07-04T18:53:40.115 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/perf 2023-07-04T18:53:40.124 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/perf 2023-07-04T18:53:40.132 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/selftest/ install 2023-07-04T18:53:40.134 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/selftest 2023-07-04T18:53:40.143 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 /var/lib/xfstests/tests/selftest 2023-07-04T18:53:40.164 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/selftest 2023-07-04T18:53:40.173 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 006.out 003.out 005.out 001.out 004.out 002.out /var/lib/xfstests/tests/selftest 2023-07-04T18:53:40.195 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/shared/ install 2023-07-04T18:53:40.196 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/shared 2023-07-04T18:53:40.205 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 002 032 298 /var/lib/xfstests/tests/shared 2023-07-04T18:53:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/shared 2023-07-04T18:53:40.227 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 032.out 298.out 002.out /var/lib/xfstests/tests/shared 2023-07-04T18:53:40.241 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/tmpfs/ install 2023-07-04T18:53:40.242 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/tmpfs 2023-07-04T18:53:40.251 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/tmpfs 2023-07-04T18:53:40.260 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/tmpfs 2023-07-04T18:53:40.268 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/tmpfs 2023-07-04T18:53:40.276 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/udf/ install 2023-07-04T18:53:40.278 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/udf 2023-07-04T18:53:40.286 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 102 /var/lib/xfstests/tests/udf 2023-07-04T18:53:40.295 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/udf 2023-07-04T18:53:40.304 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 102.out /var/lib/xfstests/tests/udf 2023-07-04T18:53:40.312 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.pA346HXQMhxfstests-dev/tests/xfs/ install 2023-07-04T18:53:40.316 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/xfs 2023-07-04T18:53:40.325 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 /var/lib/xfstests/tests/xfs 2023-07-04T18:53:42.180 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/xfs 2023-07-04T18:53:42.189 INFO:teuthology.orchestra.run.smithi077.stdout:../../install-sh -o root -g root -m 644 096.out.internal 033.out.crc 096.out.external 033.out.default 071.out.64 216.out.classic 216.out.64mblog 071.out.32 216.cfg 033.cfg 040.out 304.out 349.out 457.out 537.out 295.out 497.out 337.out 385.out 082.out 270.out 460.out 149.out 759.out 502.out 088.out 366.out 202.out 137.out 746.out 110.out 468.out 018.out 593.out 571.out 567.out 592.out 282.out 267.out 728.out 178.out 799.out 528.out 769.out 428.out 507.out 407.out 228.out 128.out 171.out 183.out 247.out 273.out 474.out 109.out 595.out 506.out 732.out 268.out 035.out 113.out 532.out 591.out 281.out 471.out 723.out 519.out 319.out 467.out 523.out 119.out 775.out 727.out 372.out 503.out 526.out 329.out 165.out 229.out 103.out 129.out 371.out 234.out 738.out 078.out 718.out 143.out 716.out 161.out 516.out 416.out 116.out 043.out 155.out 338.out 067.out 238.out 500.out 138.out 189.out 263.out 277.out 181.out 160.out 483.out 274.out 722.out 061.out 239.out 047.out 772.out 258.out 379.out 322.out 039.out 173.out 222.out 742.out 098.out 426.out 290.out 480.out 060.out 406.out 306.out 141.out 106.out 006.out 167.out 191.out 017.out 542.out 211.out 783.out 485.out 442.out 099.out 177.out 174.out 156.out 771.out 324.out 224.out 313.out 124.out 758.out 278.out 058.out 741.out 384.out 773.out 042.out 193.out 392.out 768.out 399.out 473.out 770.out 709.out 755.out 364.out 209.out 441.out 440.out 157.out 740.out 356.out 197.out 341.out 187.out 094.out 241.out 501.out 089.out 540.out 365.out 514.out 414.out 314.out 214.out 381.out 056.out 041.out 340.out 793.out 358.out 572.out 568.out 586.out 240.out 549.out 092.out 439.out 434.out 140.out 708.out 265.out 508.out 408.out 308.out 208.out 737.out 204.out 535.out 104.out 733.out 544.out 059.out 249.out 335.out 142.out 235.out 575.out 253.out 715.out 533.out 264.out 431.out 784.out 433.out 316.out 169.out 579.out 297.out 393.out 333.out 476.out 087.out 199.out 739.out 068.out 233.out 752.out 736.out 443.out 086.out 053.out 749.out 279.out 276.out 711.out 164.out 589.out 587.out 339.out 577.out 294.out 352.out 458.out 498.out 489.out 139.out 774.out 790.out 552.out 781.out 357.out 065.out 186.out 452.out 069.out 252.out 585.out 751.out 724.out 206.out 190.out 794.out 348.out 184.out 435.out 466.out 248.out 725.out 785.out 482.out 750.out 525.out 496.out 325.out 176.out 225.out 125.out 789.out 192.out 296.out 152.out 563.out 246.out 551.out 436.out 792.out 075.out 472.out 578.out 463.out 550.out 052.out 534.out 351.out 430.out 583.out 305.out 451.out 425.out 450.out 744.out 251.out 444.out 487.out 762.out 465.out 395.out 515.out 415.out 389.out 315.out 215.out 115.out 015.out 027.out 250.out 344.out 046.out 051.out 536.out 254.out 030.out 244.out 336.out 028.out 150.out 144.out 505.out 342.out 136.out 005.out 036.out 198.out 008.out 390.out 405.out 588.out 205.out 105.out 367.out 291.out 437.out 719.out 323.out 363.out 562.out 382.out 570.out 388.out 527.out 158.out 766.out 019.out 237.out 151.out 760.out 227.out 163.out 332.out 796.out 232.out 478.out 584.out 462.out 220.out 412.out 132.out 179.out 037.out 293.out 492.out 499.out 112.out 748.out 020.out 786.out 063.out 561.out 076.out 044.out 014.out 362.out 387.out 594.out 438.out 721.out 289.out 574.out 284.out 548.out 299.out 419.out 521.out 581.out 461.out 477.out 307.out 726.out 107.out 007.out 448.out 262.out 380.out 093.out 479.out 547.out 326.out 226.out 345.out 126.out 025.out 121.out 447.out 761.out 580.out 013.out 347.out 275.out 090.out 361.out 564.out 788.out 066.out 455.out 432.out 747.out 558.out 162.out 147.out 288.out 050.out 396.out 576.out 350.out 354.out 298.out 301.out 360.out 543.out 261.out 734.out 045.out 292.out 255.out 729.out 378.out 346.out 048.out 062.out 529.out 231.out 343.out 486.out 223.out 123.out 023.out 459.out 032.out 798.out 146.out 398.out 168.out 427.out 590.out 280.out 470.out 484.out 491.out 767.out 016.out 488.out 731.out 782.out 374.out 285.out 127.out 328.out 260.out 359.out 531.out 376.out 101.out 266.out 331.out 791.out 743.out 556.out 713.out 024.out 513.out 131.out 213.out 730.out 095.out 031.out 565.out 182.out 272.out 530.out 754.out 573.out 454.out 259.out 449.out 327.out 330.out 777.out 493.out 010.out 287.out 230.out 166.out 130.out 219.out 073.out 370.out 135.out 303.out 203.out 003.out 355.out 386.out 560.out 055.out 245.out 243.out 522.out 787.out 745.out 422.out 394.out 159.out 539.out 524.out 122.out 022.out 424.out 779.out 490.out 795.out 353.out 423.out 134.out 154.out 764.out 271.out 518.out 418.out 318.out 077.out 218.out 118.out 757.out 377.out 172.out 034.out 079.out 469.out 054.out 049.out 765.out 021.out 002.out 456.out 180.out 421.out 712.out 403.out 321.out 057.out 512.out 494.out 221.out 312.out 212.out 538.out 256.out 012.out 195.out 283.out 554.out 397.out 185.out 373.out 720.out 481.out 196.out 520.out 242.out 026.out 420.out 188.out 320.out 555.out 120.out 153.out 402.out 302.out 102.out 717.out 517.out 429.out 417.out 317.out 217.out 117.out 475.out 072.out 778.out 097.out 038.out 084.out 369.out 511.out 148.out 411.out 029.out 311.out 368.out 111.out 011.out 257.out 194.out 170.out 133.out 286.out 756.out 175.out 545.out 714.out 559.out 710.out 510.out 413.out 410.out 310.out 210.out 401.out 201.out 001.out 495.out 114.out 753.out 074.out 145.out 797.out 464.out 091.out 081.out 083.out 800.out 064.out 400.out 300.out 200.out 100.out 375.out 541.out 445.out 269.out 546.out 780.out 553.out 763.out 446.out 080.out 070.out 509.out 409.out 309.out 207.out 108.out 009.out 391.out 004.out 557.out 453.out 236.out 566.out 582.out 735.out 383.out 085.out 334.out 504.out 404.out 776.out /var/lib/xfstests/tests/xfs 2023-07-04T18:53:44.066 INFO:teuthology.orchestra.run.smithi077.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests 2023-07-04T18:53:44.075 INFO:teuthology.orchestra.run.smithi077.stdout:./install-sh -o root -g root -m 755 -d /var/lib/xfstests 2023-07-04T18:53:44.084 INFO:teuthology.orchestra.run.smithi077.stdout:./install-sh -o root -g root -m 755 check /var/lib/xfstests 2023-07-04T18:53:44.092 INFO:teuthology.orchestra.run.smithi077.stdout:./install-sh -o root -g root -m 644 randomize.awk /var/lib/xfstests 2023-07-04T18:53:44.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T18:53:44.106 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec ls tests/generic) 2023-07-04T18:53:44.163 INFO:teuthology.orchestra.run.smithi077.stdout:001 2023-07-04T18:53:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:001.out 2023-07-04T18:53:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:002 2023-07-04T18:53:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:002.out 2023-07-04T18:53:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:003 2023-07-04T18:53:44.164 INFO:teuthology.orchestra.run.smithi077.stdout:003.out 2023-07-04T18:53:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:004 2023-07-04T18:53:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:004.out 2023-07-04T18:53:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:005 2023-07-04T18:53:44.165 INFO:teuthology.orchestra.run.smithi077.stdout:005.out 2023-07-04T18:53:44.166 INFO:teuthology.orchestra.run.smithi077.stdout:006 2023-07-04T18:53:44.167 INFO:teuthology.orchestra.run.smithi077.stdout:006.out 2023-07-04T18:53:44.167 INFO:teuthology.orchestra.run.smithi077.stdout:007 2023-07-04T18:53:44.167 INFO:teuthology.orchestra.run.smithi077.stdout:007.out 2023-07-04T18:53:44.167 INFO:teuthology.orchestra.run.smithi077.stdout:008 2023-07-04T18:53:44.168 INFO:teuthology.orchestra.run.smithi077.stdout:008.out 2023-07-04T18:53:44.168 INFO:teuthology.orchestra.run.smithi077.stdout:009 2023-07-04T18:53:44.169 INFO:teuthology.orchestra.run.smithi077.stdout:009.out 2023-07-04T18:53:44.169 INFO:teuthology.orchestra.run.smithi077.stdout:010 2023-07-04T18:53:44.169 INFO:teuthology.orchestra.run.smithi077.stdout:010.out 2023-07-04T18:53:44.169 INFO:teuthology.orchestra.run.smithi077.stdout:011 2023-07-04T18:53:44.170 INFO:teuthology.orchestra.run.smithi077.stdout:011.out 2023-07-04T18:53:44.170 INFO:teuthology.orchestra.run.smithi077.stdout:012 2023-07-04T18:53:44.170 INFO:teuthology.orchestra.run.smithi077.stdout:012.out 2023-07-04T18:53:44.170 INFO:teuthology.orchestra.run.smithi077.stdout:013 2023-07-04T18:53:44.171 INFO:teuthology.orchestra.run.smithi077.stdout:013.out 2023-07-04T18:53:44.173 INFO:teuthology.orchestra.run.smithi077.stdout:014 2023-07-04T18:53:44.173 INFO:teuthology.orchestra.run.smithi077.stdout:014.out 2023-07-04T18:53:44.173 INFO:teuthology.orchestra.run.smithi077.stdout:015 2023-07-04T18:53:44.173 INFO:teuthology.orchestra.run.smithi077.stdout:015.out 2023-07-04T18:53:44.174 INFO:teuthology.orchestra.run.smithi077.stdout:016 2023-07-04T18:53:44.174 INFO:teuthology.orchestra.run.smithi077.stdout:016.out 2023-07-04T18:53:44.174 INFO:teuthology.orchestra.run.smithi077.stdout:017 2023-07-04T18:53:44.174 INFO:teuthology.orchestra.run.smithi077.stdout:017.out 2023-07-04T18:53:44.175 INFO:teuthology.orchestra.run.smithi077.stdout:018 2023-07-04T18:53:44.175 INFO:teuthology.orchestra.run.smithi077.stdout:018.out 2023-07-04T18:53:44.176 INFO:teuthology.orchestra.run.smithi077.stdout:019 2023-07-04T18:53:44.176 INFO:teuthology.orchestra.run.smithi077.stdout:019.out 2023-07-04T18:53:44.176 INFO:teuthology.orchestra.run.smithi077.stdout:020 2023-07-04T18:53:44.177 INFO:teuthology.orchestra.run.smithi077.stdout:020.out 2023-07-04T18:53:44.177 INFO:teuthology.orchestra.run.smithi077.stdout:021 2023-07-04T18:53:44.177 INFO:teuthology.orchestra.run.smithi077.stdout:021.out 2023-07-04T18:53:44.177 INFO:teuthology.orchestra.run.smithi077.stdout:022 2023-07-04T18:53:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:022.out 2023-07-04T18:53:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:023 2023-07-04T18:53:44.178 INFO:teuthology.orchestra.run.smithi077.stdout:023.out 2023-07-04T18:53:44.179 INFO:teuthology.orchestra.run.smithi077.stdout:024 2023-07-04T18:53:44.179 INFO:teuthology.orchestra.run.smithi077.stdout:024.out 2023-07-04T18:53:44.179 INFO:teuthology.orchestra.run.smithi077.stdout:025 2023-07-04T18:53:44.180 INFO:teuthology.orchestra.run.smithi077.stdout:025.out 2023-07-04T18:53:44.180 INFO:teuthology.orchestra.run.smithi077.stdout:026 2023-07-04T18:53:44.180 INFO:teuthology.orchestra.run.smithi077.stdout:026.out 2023-07-04T18:53:44.180 INFO:teuthology.orchestra.run.smithi077.stdout:027 2023-07-04T18:53:44.181 INFO:teuthology.orchestra.run.smithi077.stdout:027.out 2023-07-04T18:53:44.181 INFO:teuthology.orchestra.run.smithi077.stdout:028 2023-07-04T18:53:44.181 INFO:teuthology.orchestra.run.smithi077.stdout:028.out 2023-07-04T18:53:44.182 INFO:teuthology.orchestra.run.smithi077.stdout:029 2023-07-04T18:53:44.182 INFO:teuthology.orchestra.run.smithi077.stdout:029.out 2023-07-04T18:53:44.183 INFO:teuthology.orchestra.run.smithi077.stdout:030 2023-07-04T18:53:44.183 INFO:teuthology.orchestra.run.smithi077.stdout:030.out 2023-07-04T18:53:44.183 INFO:teuthology.orchestra.run.smithi077.stdout:031 2023-07-04T18:53:44.184 INFO:teuthology.orchestra.run.smithi077.stdout:031.out 2023-07-04T18:53:44.184 INFO:teuthology.orchestra.run.smithi077.stdout:032 2023-07-04T18:53:44.184 INFO:teuthology.orchestra.run.smithi077.stdout:032.out 2023-07-04T18:53:44.185 INFO:teuthology.orchestra.run.smithi077.stdout:033 2023-07-04T18:53:44.185 INFO:teuthology.orchestra.run.smithi077.stdout:033.out 2023-07-04T18:53:44.185 INFO:teuthology.orchestra.run.smithi077.stdout:034 2023-07-04T18:53:44.185 INFO:teuthology.orchestra.run.smithi077.stdout:034.out 2023-07-04T18:53:44.185 INFO:teuthology.orchestra.run.smithi077.stdout:035 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:035.cfg 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:035.out.default 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:035.out.nfs 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:036 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:036.out 2023-07-04T18:53:44.186 INFO:teuthology.orchestra.run.smithi077.stdout:037 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:037.out 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:038 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:038.out 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:039 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:039.out 2023-07-04T18:53:44.187 INFO:teuthology.orchestra.run.smithi077.stdout:040 2023-07-04T18:53:44.188 INFO:teuthology.orchestra.run.smithi077.stdout:040.out 2023-07-04T18:53:44.188 INFO:teuthology.orchestra.run.smithi077.stdout:041 2023-07-04T18:53:44.188 INFO:teuthology.orchestra.run.smithi077.stdout:041.out 2023-07-04T18:53:44.188 INFO:teuthology.orchestra.run.smithi077.stdout:042 2023-07-04T18:53:44.188 INFO:teuthology.orchestra.run.smithi077.stdout:042.out 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:043 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:043.out 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:044 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:044.out 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:045 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:045.out 2023-07-04T18:53:44.189 INFO:teuthology.orchestra.run.smithi077.stdout:046 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:046.out 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:047 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:047.out 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:048 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:048.out 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:049 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:049.out 2023-07-04T18:53:44.190 INFO:teuthology.orchestra.run.smithi077.stdout:050 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:050.cfg 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:050.out.default 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:050.out.nojournal 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:050.out.xfsquota 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:051 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:051.out 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:052 2023-07-04T18:53:44.191 INFO:teuthology.orchestra.run.smithi077.stdout:052.out 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:053 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:053.out 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:054 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:054.out 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:055 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:055.out 2023-07-04T18:53:44.192 INFO:teuthology.orchestra.run.smithi077.stdout:056 2023-07-04T18:53:44.193 INFO:teuthology.orchestra.run.smithi077.stdout:056.out 2023-07-04T18:53:44.193 INFO:teuthology.orchestra.run.smithi077.stdout:057 2023-07-04T18:53:44.193 INFO:teuthology.orchestra.run.smithi077.stdout:057.out 2023-07-04T18:53:44.194 INFO:teuthology.orchestra.run.smithi077.stdout:058 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:058.out 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:059 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:059.out 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:060 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:060.out 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:061 2023-07-04T18:53:44.195 INFO:teuthology.orchestra.run.smithi077.stdout:061.out 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:062 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:062.out 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:063 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:063.out 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:064 2023-07-04T18:53:44.196 INFO:teuthology.orchestra.run.smithi077.stdout:064.out 2023-07-04T18:53:44.197 INFO:teuthology.orchestra.run.smithi077.stdout:065 2023-07-04T18:53:44.197 INFO:teuthology.orchestra.run.smithi077.stdout:065.out 2023-07-04T18:53:44.197 INFO:teuthology.orchestra.run.smithi077.stdout:066 2023-07-04T18:53:44.197 INFO:teuthology.orchestra.run.smithi077.stdout:066.out 2023-07-04T18:53:44.197 INFO:teuthology.orchestra.run.smithi077.stdout:067 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:067.out 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:068 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:068.out 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:069 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:069.out 2023-07-04T18:53:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:070 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:070.out 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:071 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:071.out 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:072 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:072.out 2023-07-04T18:53:44.199 INFO:teuthology.orchestra.run.smithi077.stdout:073 2023-07-04T18:53:44.200 INFO:teuthology.orchestra.run.smithi077.stdout:073.out 2023-07-04T18:53:44.200 INFO:teuthology.orchestra.run.smithi077.stdout:074 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:074.out 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:075 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:075.out 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:076 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:076.out 2023-07-04T18:53:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:077 2023-07-04T18:53:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:077.out 2023-07-04T18:53:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:078 2023-07-04T18:53:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:078.out 2023-07-04T18:53:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:079 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:079.out 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:080 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:080.out 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:081 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:081.out 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:082 2023-07-04T18:53:44.203 INFO:teuthology.orchestra.run.smithi077.stdout:082.out 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:083 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:083.out 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:084 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:084.out 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:085 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:085.out 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:086 2023-07-04T18:53:44.204 INFO:teuthology.orchestra.run.smithi077.stdout:086.out 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:087 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:087.out 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:088 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:088.out 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:089 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:089.out 2023-07-04T18:53:44.205 INFO:teuthology.orchestra.run.smithi077.stdout:090 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:090.out 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:091 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:091.out 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:092 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:092.out 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:093 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:093.out 2023-07-04T18:53:44.206 INFO:teuthology.orchestra.run.smithi077.stdout:094 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:094.out 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:095 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:095.out 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:096 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:096.out 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:097 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:097.out 2023-07-04T18:53:44.207 INFO:teuthology.orchestra.run.smithi077.stdout:098 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:098.out 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:099 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:099.out 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:100 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:100.out 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:101 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:101.out 2023-07-04T18:53:44.208 INFO:teuthology.orchestra.run.smithi077.stdout:102 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:102.out 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:103 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:103.out 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:104 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:104.out 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:105 2023-07-04T18:53:44.209 INFO:teuthology.orchestra.run.smithi077.stdout:105.out 2023-07-04T18:53:44.210 INFO:teuthology.orchestra.run.smithi077.stdout:106 2023-07-04T18:53:44.210 INFO:teuthology.orchestra.run.smithi077.stdout:106.out 2023-07-04T18:53:44.210 INFO:teuthology.orchestra.run.smithi077.stdout:107 2023-07-04T18:53:44.210 INFO:teuthology.orchestra.run.smithi077.stdout:107.out 2023-07-04T18:53:44.211 INFO:teuthology.orchestra.run.smithi077.stdout:108 2023-07-04T18:53:44.211 INFO:teuthology.orchestra.run.smithi077.stdout:108.out 2023-07-04T18:53:44.211 INFO:teuthology.orchestra.run.smithi077.stdout:109 2023-07-04T18:53:44.211 INFO:teuthology.orchestra.run.smithi077.stdout:109.out 2023-07-04T18:53:44.211 INFO:teuthology.orchestra.run.smithi077.stdout:110 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:110.out 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:111 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:111.out 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:112 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:112.out 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:113 2023-07-04T18:53:44.212 INFO:teuthology.orchestra.run.smithi077.stdout:113.out 2023-07-04T18:53:44.213 INFO:teuthology.orchestra.run.smithi077.stdout:114 2023-07-04T18:53:44.213 INFO:teuthology.orchestra.run.smithi077.stdout:114.out 2023-07-04T18:53:44.213 INFO:teuthology.orchestra.run.smithi077.stdout:115 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:115.out 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:116 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:116.out 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:117 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:117.out 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:118 2023-07-04T18:53:44.214 INFO:teuthology.orchestra.run.smithi077.stdout:118.out 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:119 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:119.out 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:120 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:120.out 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:121 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:121.out 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:122 2023-07-04T18:53:44.215 INFO:teuthology.orchestra.run.smithi077.stdout:122.out 2023-07-04T18:53:44.216 INFO:teuthology.orchestra.run.smithi077.stdout:123 2023-07-04T18:53:44.216 INFO:teuthology.orchestra.run.smithi077.stdout:123.out 2023-07-04T18:53:44.216 INFO:teuthology.orchestra.run.smithi077.stdout:124 2023-07-04T18:53:44.216 INFO:teuthology.orchestra.run.smithi077.stdout:124.out 2023-07-04T18:53:44.216 INFO:teuthology.orchestra.run.smithi077.stdout:125 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:125.out 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:126 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:126.out 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:127 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:127.out 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:128 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:128.out 2023-07-04T18:53:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:129 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:129.out 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:130 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:130.out 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:131 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:131.out 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:132 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:132.out 2023-07-04T18:53:44.218 INFO:teuthology.orchestra.run.smithi077.stdout:133 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:133.out 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:134 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:134.out 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:135 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:135.out 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:136 2023-07-04T18:53:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:136.out 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:137 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:137.out 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:138 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:138.out 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:139 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:139.out 2023-07-04T18:53:44.220 INFO:teuthology.orchestra.run.smithi077.stdout:140 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:140.out 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:141 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:141.out 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:142 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:142.out 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:143 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:143.out 2023-07-04T18:53:44.221 INFO:teuthology.orchestra.run.smithi077.stdout:144 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:144.out 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:145 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:145.out 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:146 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:146.out 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:147 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:147.out 2023-07-04T18:53:44.222 INFO:teuthology.orchestra.run.smithi077.stdout:148 2023-07-04T18:53:44.223 INFO:teuthology.orchestra.run.smithi077.stdout:148.out 2023-07-04T18:53:44.223 INFO:teuthology.orchestra.run.smithi077.stdout:149 2023-07-04T18:53:44.223 INFO:teuthology.orchestra.run.smithi077.stdout:149.out 2023-07-04T18:53:44.223 INFO:teuthology.orchestra.run.smithi077.stdout:150 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:150.out 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:151 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:151.out 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:152 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:152.out 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:153 2023-07-04T18:53:44.224 INFO:teuthology.orchestra.run.smithi077.stdout:153.out 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:154 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:154.out 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:155 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:155.out 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:156 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:156.out 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:157 2023-07-04T18:53:44.225 INFO:teuthology.orchestra.run.smithi077.stdout:157.out 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:158 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:158.out 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:159 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:159.out 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:160 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:160.out 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:161 2023-07-04T18:53:44.226 INFO:teuthology.orchestra.run.smithi077.stdout:161.out 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:162 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:162.out 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:163 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:163.out 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:164 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:164.out 2023-07-04T18:53:44.227 INFO:teuthology.orchestra.run.smithi077.stdout:165 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:165.out 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:166 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:166.out 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:167 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:167.out 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:168 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:168.out 2023-07-04T18:53:44.228 INFO:teuthology.orchestra.run.smithi077.stdout:169 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:169.out 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:170 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:170.out 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:171 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:171.out 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:172 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:172.out 2023-07-04T18:53:44.229 INFO:teuthology.orchestra.run.smithi077.stdout:173 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:173.out 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:174 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:174.out 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:175 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:175.out 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:176 2023-07-04T18:53:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:176.out 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:177 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:177.out 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:178 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:178.out 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:179 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:179.out 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:180 2023-07-04T18:53:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:180.out 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:181 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:181.out 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:182 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:182.out 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:183 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:183.out 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:184 2023-07-04T18:53:44.232 INFO:teuthology.orchestra.run.smithi077.stdout:184.out 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:185 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:185.out 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:186 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:186.out 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:187 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:187.out 2023-07-04T18:53:44.233 INFO:teuthology.orchestra.run.smithi077.stdout:188 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:188.out 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:189 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:189.out 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:190 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:190.out 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:191 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:191.out 2023-07-04T18:53:44.234 INFO:teuthology.orchestra.run.smithi077.stdout:192 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:192.out 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:193 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:193.out 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:194 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:194.out 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:195 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:195.out 2023-07-04T18:53:44.235 INFO:teuthology.orchestra.run.smithi077.stdout:196 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:196.out 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:197 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:197.out 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:198 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:198.out 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:199 2023-07-04T18:53:44.236 INFO:teuthology.orchestra.run.smithi077.stdout:199.out 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:200 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:200.out 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:201 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:201.out 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:202 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:202.out 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:203 2023-07-04T18:53:44.237 INFO:teuthology.orchestra.run.smithi077.stdout:203.out 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:204 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:204.out 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:205 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:205.out 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:206 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:206.out 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:207 2023-07-04T18:53:44.238 INFO:teuthology.orchestra.run.smithi077.stdout:207.out 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:208 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:208.out 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:209 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:209.out 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:210 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:210.out 2023-07-04T18:53:44.239 INFO:teuthology.orchestra.run.smithi077.stdout:211 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:211.out 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:212 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:212.out 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:213 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:213.out 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:214 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:214.out 2023-07-04T18:53:44.240 INFO:teuthology.orchestra.run.smithi077.stdout:215 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:215.out 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:216 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:216.out 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:217 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:217.out 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:218 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:218.out 2023-07-04T18:53:44.241 INFO:teuthology.orchestra.run.smithi077.stdout:219 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:219.out 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:220 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:220.out 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:221 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:221.out 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:222 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:222.out 2023-07-04T18:53:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:223 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:223.out 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:224 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:224.out 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:225 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:225.out 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:226 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:226.out 2023-07-04T18:53:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:227 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:227.out 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:228 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:228.out 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:229 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:229.out 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:230 2023-07-04T18:53:44.244 INFO:teuthology.orchestra.run.smithi077.stdout:230.out 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:231 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:231.out 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:232 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:232.out 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:233 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:233.out 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:234 2023-07-04T18:53:44.245 INFO:teuthology.orchestra.run.smithi077.stdout:234.out 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:235 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:235.out 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:236 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:236.out 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:237 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:237.out 2023-07-04T18:53:44.246 INFO:teuthology.orchestra.run.smithi077.stdout:238 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:238.out 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:239 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:239.out 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:240 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:240.out 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:241 2023-07-04T18:53:44.247 INFO:teuthology.orchestra.run.smithi077.stdout:241.out 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:242 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:242.out 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:243 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:243.out 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:244 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:244.out 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:245 2023-07-04T18:53:44.248 INFO:teuthology.orchestra.run.smithi077.stdout:245.out 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:246 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:246.out 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:247 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:247.out 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:248 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:248.out 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:249 2023-07-04T18:53:44.249 INFO:teuthology.orchestra.run.smithi077.stdout:249.out 2023-07-04T18:53:44.250 INFO:teuthology.orchestra.run.smithi077.stdout:250 2023-07-04T18:53:44.250 INFO:teuthology.orchestra.run.smithi077.stdout:250.out 2023-07-04T18:53:44.250 INFO:teuthology.orchestra.run.smithi077.stdout:251 2023-07-04T18:53:44.250 INFO:teuthology.orchestra.run.smithi077.stdout:251.out 2023-07-04T18:53:44.250 INFO:teuthology.orchestra.run.smithi077.stdout:252 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:252.out 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:253 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:253.out 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:254 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:254.out 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:255 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:255.out 2023-07-04T18:53:44.251 INFO:teuthology.orchestra.run.smithi077.stdout:256 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:256.out 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:257 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:257.out 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:258 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:258.out 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:259 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:259.out 2023-07-04T18:53:44.252 INFO:teuthology.orchestra.run.smithi077.stdout:260 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:260.out 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:261 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:261.out 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:262 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:262.out 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:263 2023-07-04T18:53:44.253 INFO:teuthology.orchestra.run.smithi077.stdout:263.out 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:264 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:264.out 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:265 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:265.out 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:266 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:266.out 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:267 2023-07-04T18:53:44.254 INFO:teuthology.orchestra.run.smithi077.stdout:267.out 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:268 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:268.out 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:269 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:269.out 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:270 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:270.out 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:271 2023-07-04T18:53:44.255 INFO:teuthology.orchestra.run.smithi077.stdout:271.out 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:272 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:272.out 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:273 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:273.out 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:274 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:274.out 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:275 2023-07-04T18:53:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:275.out 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:276 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:276.out 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:277 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:277.out 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:278 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:278.out 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:279 2023-07-04T18:53:44.257 INFO:teuthology.orchestra.run.smithi077.stdout:279.out 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:280 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:280.out 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:281 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:281.out 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:282 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:282.out 2023-07-04T18:53:44.258 INFO:teuthology.orchestra.run.smithi077.stdout:283 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:283.out 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:284 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:284.out 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:285 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:285.out 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:286 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:286.out 2023-07-04T18:53:44.259 INFO:teuthology.orchestra.run.smithi077.stdout:287 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:287.out 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:288 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:288.out 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:289 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:289.out 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:290 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:290.out 2023-07-04T18:53:44.260 INFO:teuthology.orchestra.run.smithi077.stdout:291 2023-07-04T18:53:44.261 INFO:teuthology.orchestra.run.smithi077.stdout:291.out 2023-07-04T18:53:44.261 INFO:teuthology.orchestra.run.smithi077.stdout:292 2023-07-04T18:53:44.261 INFO:teuthology.orchestra.run.smithi077.stdout:292.out 2023-07-04T18:53:44.261 INFO:teuthology.orchestra.run.smithi077.stdout:293 2023-07-04T18:53:44.261 INFO:teuthology.orchestra.run.smithi077.stdout:293.out 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:294 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:294.out 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:295 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:295.out 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:296 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:296.out 2023-07-04T18:53:44.262 INFO:teuthology.orchestra.run.smithi077.stdout:297 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:297.out 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:298 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:298.out 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:299 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:299.out 2023-07-04T18:53:44.263 INFO:teuthology.orchestra.run.smithi077.stdout:300 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:300.out 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:301 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:301.out 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:302 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:302.out 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:303 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:303.out 2023-07-04T18:53:44.264 INFO:teuthology.orchestra.run.smithi077.stdout:304 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:304.out 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:305 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:305.out 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:306 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:306.out 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:307 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:307.out 2023-07-04T18:53:44.265 INFO:teuthology.orchestra.run.smithi077.stdout:308 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:308.out 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:309 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:309.out 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:310 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:310.out 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:311 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:311.out 2023-07-04T18:53:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:312 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:312.out 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:313 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:313.out 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:314 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:314.out 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:315 2023-07-04T18:53:44.267 INFO:teuthology.orchestra.run.smithi077.stdout:315.out 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:316 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:316.out 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:317 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:317.out 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:318 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:318.out 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:319 2023-07-04T18:53:44.268 INFO:teuthology.orchestra.run.smithi077.stdout:319.out 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:320 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:320.out 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:321 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:321.out 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:322 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:322.out 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:323 2023-07-04T18:53:44.269 INFO:teuthology.orchestra.run.smithi077.stdout:323.out 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:324 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:324.out 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:325 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:325.out 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:326 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:326.out 2023-07-04T18:53:44.270 INFO:teuthology.orchestra.run.smithi077.stdout:327 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:327.out 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:328 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:328.out 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:329 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:329.out 2023-07-04T18:53:44.271 INFO:teuthology.orchestra.run.smithi077.stdout:330 2023-07-04T18:53:44.272 INFO:teuthology.orchestra.run.smithi077.stdout:330.out 2023-07-04T18:53:44.272 INFO:teuthology.orchestra.run.smithi077.stdout:331 2023-07-04T18:53:44.272 INFO:teuthology.orchestra.run.smithi077.stdout:331.out 2023-07-04T18:53:44.272 INFO:teuthology.orchestra.run.smithi077.stdout:332 2023-07-04T18:53:44.272 INFO:teuthology.orchestra.run.smithi077.stdout:332.out 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:333 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:333.out 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:334 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:334.out 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:335 2023-07-04T18:53:44.273 INFO:teuthology.orchestra.run.smithi077.stdout:335.out 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:336 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:336.out 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:337 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:337.out 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:338 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:338.out 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:339 2023-07-04T18:53:44.274 INFO:teuthology.orchestra.run.smithi077.stdout:339.out 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:340 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:340.out 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:341 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:341.out 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:342 2023-07-04T18:53:44.275 INFO:teuthology.orchestra.run.smithi077.stdout:342.out 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:343 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:343.out 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:344 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:344.out 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:345 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:345.out 2023-07-04T18:53:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:346 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:346.out 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:347 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:347.out 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:348 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:348.out 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:349 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:349.out 2023-07-04T18:53:44.277 INFO:teuthology.orchestra.run.smithi077.stdout:350 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:350.out 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:351 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:351.out 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:352 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:352.out 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:353 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:353.out 2023-07-04T18:53:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:354 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:354.out 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:355 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:355.out 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:356 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:356.out 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:357 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:357.out 2023-07-04T18:53:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:358 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:358.out 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:359 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:359.out 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:360 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:360.out 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:361 2023-07-04T18:53:44.280 INFO:teuthology.orchestra.run.smithi077.stdout:361.out 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:362 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:362.out 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:363 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:363.out 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:364 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:364.out 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:365 2023-07-04T18:53:44.281 INFO:teuthology.orchestra.run.smithi077.stdout:365.out 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:366 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:366.out 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:367 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:367.out 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:368 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:368.out 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:369 2023-07-04T18:53:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:369.out 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:370 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:370.out 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:371 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:371.out 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:372 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:372.out 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:373 2023-07-04T18:53:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:373.out 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:374 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:374.out 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:375 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:375.out 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:376 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:376.out 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:377 2023-07-04T18:53:44.284 INFO:teuthology.orchestra.run.smithi077.stdout:377.out 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:378 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:378.out 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:379 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:379.out 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:380 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:380.out 2023-07-04T18:53:44.285 INFO:teuthology.orchestra.run.smithi077.stdout:381 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:381.out 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:382 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:382.out 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:383 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:383.out 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:384 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:384.out 2023-07-04T18:53:44.286 INFO:teuthology.orchestra.run.smithi077.stdout:385 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:385.out 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:386 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:386.out 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:387 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:387.out 2023-07-04T18:53:44.287 INFO:teuthology.orchestra.run.smithi077.stdout:388 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:388.out 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:389 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:389.out 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:390 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:390.out 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:391 2023-07-04T18:53:44.288 INFO:teuthology.orchestra.run.smithi077.stdout:391.out 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:392 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:392.out 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:393 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:393.out 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:394 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:394.out 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:395 2023-07-04T18:53:44.289 INFO:teuthology.orchestra.run.smithi077.stdout:395.out 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:396 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:396.out 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:397 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:397.out 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:398 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:398.out 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:399 2023-07-04T18:53:44.290 INFO:teuthology.orchestra.run.smithi077.stdout:399.out 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:400 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:400.out 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:401 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:401.out 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:402 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:402.out 2023-07-04T18:53:44.291 INFO:teuthology.orchestra.run.smithi077.stdout:403 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:403.out 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:404 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:404.out 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:405 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:405.out 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:406 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:406.out 2023-07-04T18:53:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:407 2023-07-04T18:53:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:407.out 2023-07-04T18:53:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:408 2023-07-04T18:53:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:408.out 2023-07-04T18:53:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:409 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:409.out 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:410 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:410.out 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:411 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:411.out 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:412 2023-07-04T18:53:44.294 INFO:teuthology.orchestra.run.smithi077.stdout:412.out 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:413 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:413.out 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:414 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:414.out 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:415 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:415.out 2023-07-04T18:53:44.295 INFO:teuthology.orchestra.run.smithi077.stdout:416 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:416.out 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:417 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:417.out 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:418 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:418.out 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:419 2023-07-04T18:53:44.296 INFO:teuthology.orchestra.run.smithi077.stdout:419.out 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:420 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:420.out 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:421 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:421.out 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:422 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:422.out 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:423 2023-07-04T18:53:44.297 INFO:teuthology.orchestra.run.smithi077.stdout:423.out 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:424 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:424.out 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:425 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:425.out 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:426 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:426.out 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:427 2023-07-04T18:53:44.298 INFO:teuthology.orchestra.run.smithi077.stdout:427.out 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:428 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:428.out 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:429 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:429.out 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:430 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:430.out 2023-07-04T18:53:44.299 INFO:teuthology.orchestra.run.smithi077.stdout:431 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:431.out 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:432 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:432.out 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:433 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:433.out 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:434 2023-07-04T18:53:44.300 INFO:teuthology.orchestra.run.smithi077.stdout:434.out 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:435 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:435.out 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:436 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:436.out 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:437 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:437.out 2023-07-04T18:53:44.301 INFO:teuthology.orchestra.run.smithi077.stdout:438 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:438.out 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:439 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:439.out 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:440 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:440.out 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:441 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:441.out 2023-07-04T18:53:44.302 INFO:teuthology.orchestra.run.smithi077.stdout:442 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:442.out 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:443 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:443.out 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:444 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:444.out 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:445 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:445.out 2023-07-04T18:53:44.303 INFO:teuthology.orchestra.run.smithi077.stdout:446 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:446.out 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:447 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:447.out 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:448 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:448.out 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:449 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:449.out 2023-07-04T18:53:44.304 INFO:teuthology.orchestra.run.smithi077.stdout:450 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:450.out 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:451 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:451.out 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:452 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:452.out 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:453 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:453.out 2023-07-04T18:53:44.305 INFO:teuthology.orchestra.run.smithi077.stdout:454 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:454.out 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:455 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:455.out 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:456 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:456.out 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:457 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:457.out 2023-07-04T18:53:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:458 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:458.out 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:459 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:459.out 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:460 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:460.out 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:461 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:461.out 2023-07-04T18:53:44.307 INFO:teuthology.orchestra.run.smithi077.stdout:462 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:462.out 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:463 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:463.out 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:464 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:464.out 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:465 2023-07-04T18:53:44.308 INFO:teuthology.orchestra.run.smithi077.stdout:465.out 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:466 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:466.out 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:467 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:467.out 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:468 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:468.out 2023-07-04T18:53:44.309 INFO:teuthology.orchestra.run.smithi077.stdout:469 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:469.out 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:470 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:470.out 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:471 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:471.out 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:472 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:472.out 2023-07-04T18:53:44.310 INFO:teuthology.orchestra.run.smithi077.stdout:473 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:473.out 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:474 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:474.out 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:475 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:475.out 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:476 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:476.out 2023-07-04T18:53:44.311 INFO:teuthology.orchestra.run.smithi077.stdout:477 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:477.out 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:478 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:478.out 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:479 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:479.out 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:480 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:480.out 2023-07-04T18:53:44.312 INFO:teuthology.orchestra.run.smithi077.stdout:481 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:481.out 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:482 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:482.out 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:483 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:483.out 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:484 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:484.out 2023-07-04T18:53:44.313 INFO:teuthology.orchestra.run.smithi077.stdout:485 2023-07-04T18:53:44.314 INFO:teuthology.orchestra.run.smithi077.stdout:485.out 2023-07-04T18:53:44.314 INFO:teuthology.orchestra.run.smithi077.stdout:486 2023-07-04T18:53:44.314 INFO:teuthology.orchestra.run.smithi077.stdout:486.out 2023-07-04T18:53:44.314 INFO:teuthology.orchestra.run.smithi077.stdout:487 2023-07-04T18:53:44.314 INFO:teuthology.orchestra.run.smithi077.stdout:487.out 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:488 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:488.out 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:489 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:489.out 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:490 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:490.out 2023-07-04T18:53:44.315 INFO:teuthology.orchestra.run.smithi077.stdout:491 2023-07-04T18:53:44.316 INFO:teuthology.orchestra.run.smithi077.stdout:491.out 2023-07-04T18:53:44.316 INFO:teuthology.orchestra.run.smithi077.stdout:492 2023-07-04T18:53:44.316 INFO:teuthology.orchestra.run.smithi077.stdout:492.out 2023-07-04T18:53:44.316 INFO:teuthology.orchestra.run.smithi077.stdout:493 2023-07-04T18:53:44.316 INFO:teuthology.orchestra.run.smithi077.stdout:493.out 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:494 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:494.out 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:495 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:495.out 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:496 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:496.out 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:497 2023-07-04T18:53:44.317 INFO:teuthology.orchestra.run.smithi077.stdout:497.out 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:498 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:498.out 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:499 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:499.out 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:500 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:500.out 2023-07-04T18:53:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:501 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:501.out 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:502 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:502.out 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:503 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:503.out 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:504 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:504.out 2023-07-04T18:53:44.319 INFO:teuthology.orchestra.run.smithi077.stdout:505 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:505.out 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:506 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:506.out 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:507 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:507.out 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:508 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:508.out 2023-07-04T18:53:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:509 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:509.out 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:510 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:510.out 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:511 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:511.out 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:512 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:512.out 2023-07-04T18:53:44.321 INFO:teuthology.orchestra.run.smithi077.stdout:513 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:513.out 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:514 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:514.out 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:515 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:515.out 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:516 2023-07-04T18:53:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:516.out 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:517 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:517.out 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:518 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:518.out 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:519 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:519.out 2023-07-04T18:53:44.323 INFO:teuthology.orchestra.run.smithi077.stdout:520 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:520.out 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:521 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:521.out 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:522 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:522.out 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:523 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:523.out 2023-07-04T18:53:44.324 INFO:teuthology.orchestra.run.smithi077.stdout:524 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:524.out 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:525 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:525.out 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:526 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:526.out 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:527 2023-07-04T18:53:44.325 INFO:teuthology.orchestra.run.smithi077.stdout:527.out 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:528 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:528.out 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:529 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:529.out 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:530 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:530.out 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:531 2023-07-04T18:53:44.326 INFO:teuthology.orchestra.run.smithi077.stdout:531.out 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:532 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:532.out 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:533 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:533.out 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:534 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:534.out 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:535 2023-07-04T18:53:44.327 INFO:teuthology.orchestra.run.smithi077.stdout:535.out 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:536 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:536.out 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:537 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:537.out 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:538 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:538.out 2023-07-04T18:53:44.328 INFO:teuthology.orchestra.run.smithi077.stdout:539 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:539.out 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:540 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:540.out 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:541 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:541.out 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:542 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:542.out 2023-07-04T18:53:44.329 INFO:teuthology.orchestra.run.smithi077.stdout:543 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:543.out 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:544 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:544.out 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:545 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:545.out 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:546 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:546.out 2023-07-04T18:53:44.330 INFO:teuthology.orchestra.run.smithi077.stdout:547 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:547.out 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:548 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:548.out 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:549 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:549.out 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:550 2023-07-04T18:53:44.331 INFO:teuthology.orchestra.run.smithi077.stdout:550.out 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:551 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:551.out 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:552 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:552.out 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:553 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:553.out 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:554 2023-07-04T18:53:44.332 INFO:teuthology.orchestra.run.smithi077.stdout:554.out 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:555 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:555.out 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:556 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:556.out 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:557 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:557.out 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:558 2023-07-04T18:53:44.333 INFO:teuthology.orchestra.run.smithi077.stdout:558.out 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:559 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:559.out 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:560 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:560.out 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:561 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:561.out 2023-07-04T18:53:44.334 INFO:teuthology.orchestra.run.smithi077.stdout:562 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:562.out 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:563 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:563.out 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:564 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:564.out 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:565 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:565.out 2023-07-04T18:53:44.335 INFO:teuthology.orchestra.run.smithi077.stdout:566 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:566.out 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:567 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:567.out 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:568 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:568.out 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:569 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:569.out 2023-07-04T18:53:44.336 INFO:teuthology.orchestra.run.smithi077.stdout:570 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:570.out 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:571 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:571.out 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:572 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:572.out 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:573 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:573.out 2023-07-04T18:53:44.337 INFO:teuthology.orchestra.run.smithi077.stdout:574 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:574.out 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:575 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:575.out 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:576 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:576.out 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:577 2023-07-04T18:53:44.338 INFO:teuthology.orchestra.run.smithi077.stdout:577.out 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:578 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:578.out 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:579 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:579.out 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:580 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:580.out 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:581 2023-07-04T18:53:44.339 INFO:teuthology.orchestra.run.smithi077.stdout:581.out 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:582 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:582.out 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:583 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:583.out 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:584 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:584.out 2023-07-04T18:53:44.340 INFO:teuthology.orchestra.run.smithi077.stdout:585 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:585.out 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:586 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:586.out 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:587 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:587.out 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:588 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:588.out 2023-07-04T18:53:44.341 INFO:teuthology.orchestra.run.smithi077.stdout:589 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:589.out 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:590 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:590.out 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:591 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:591.out 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:592 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:592.out 2023-07-04T18:53:44.342 INFO:teuthology.orchestra.run.smithi077.stdout:593 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:593.out 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:594 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:594.out 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:595 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:595.out 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:596 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:596.out 2023-07-04T18:53:44.343 INFO:teuthology.orchestra.run.smithi077.stdout:597 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:597.out 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:598 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:598.out 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:599 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:599.out 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:600 2023-07-04T18:53:44.344 INFO:teuthology.orchestra.run.smithi077.stdout:600.out 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:601 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:601.out 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:602 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:602.out 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:603 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:603.out 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:604 2023-07-04T18:53:44.345 INFO:teuthology.orchestra.run.smithi077.stdout:604.out 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:605 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:605.out 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:606 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:606.out 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:607 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:607.out 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:608 2023-07-04T18:53:44.346 INFO:teuthology.orchestra.run.smithi077.stdout:608.out 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:609 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:609.out 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:610 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:610.out 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:611 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:611.out 2023-07-04T18:53:44.347 INFO:teuthology.orchestra.run.smithi077.stdout:612 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:612.out 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:613 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:613.out 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:614 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:614.out 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:615 2023-07-04T18:53:44.348 INFO:teuthology.orchestra.run.smithi077.stdout:615.out 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:616 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:616.out 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:617 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:617.out 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:618 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:618.out 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:619 2023-07-04T18:53:44.349 INFO:teuthology.orchestra.run.smithi077.stdout:619.out 2023-07-04T18:53:44.350 INFO:teuthology.orchestra.run.smithi077.stdout:620 2023-07-04T18:53:44.350 INFO:teuthology.orchestra.run.smithi077.stdout:620.out 2023-07-04T18:53:44.350 INFO:teuthology.orchestra.run.smithi077.stdout:621 2023-07-04T18:53:44.350 INFO:teuthology.orchestra.run.smithi077.stdout:621.out 2023-07-04T18:53:44.350 INFO:teuthology.orchestra.run.smithi077.stdout:622 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:622.out 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:623 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:623.out 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:624 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:624.out 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:625 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:625.out 2023-07-04T18:53:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:626 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:626.out 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:627 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:627.out 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:628 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:628.out 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:629 2023-07-04T18:53:44.352 INFO:teuthology.orchestra.run.smithi077.stdout:629.out 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:630 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:630.out 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:631 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:631.out 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:632 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:632.out 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:633 2023-07-04T18:53:44.353 INFO:teuthology.orchestra.run.smithi077.stdout:633.out 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:634 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:634.out 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:635 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:635.out 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:636 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:636.out 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:637 2023-07-04T18:53:44.354 INFO:teuthology.orchestra.run.smithi077.stdout:637.out 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:638 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:638.out 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:639 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:639.out 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:640 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:640.out 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:641 2023-07-04T18:53:44.355 INFO:teuthology.orchestra.run.smithi077.stdout:641.out 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:642 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:642.out 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:643 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:643.out 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:644 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:644.out 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:645 2023-07-04T18:53:44.356 INFO:teuthology.orchestra.run.smithi077.stdout:645.out 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:646 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:646.out 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:647 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:647.out 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:648 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:648.out 2023-07-04T18:53:44.357 INFO:teuthology.orchestra.run.smithi077.stdout:649 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:649.out 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:650 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:650.out 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:651 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:651.out 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:652 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:652.out 2023-07-04T18:53:44.358 INFO:teuthology.orchestra.run.smithi077.stdout:653 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:653.out 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:654 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:654.out 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:655 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:655.out 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:656 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:656.out 2023-07-04T18:53:44.359 INFO:teuthology.orchestra.run.smithi077.stdout:657 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:657.out 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:658 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:658.out 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:659 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:659.out 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:660 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:660.out 2023-07-04T18:53:44.360 INFO:teuthology.orchestra.run.smithi077.stdout:661 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:661.out 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:662 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:662.out 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:663 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:663.out 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:664 2023-07-04T18:53:44.361 INFO:teuthology.orchestra.run.smithi077.stdout:664.out 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:665 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:665.out 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:666 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:666.out 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:667 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:667.out 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:668 2023-07-04T18:53:44.362 INFO:teuthology.orchestra.run.smithi077.stdout:668.out 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:669 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:669.out 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:670 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:670.out 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:671 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:671.out 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:672 2023-07-04T18:53:44.363 INFO:teuthology.orchestra.run.smithi077.stdout:672.out 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:673 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:673.out 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:674 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:674.out 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:675 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:675.out 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:676 2023-07-04T18:53:44.364 INFO:teuthology.orchestra.run.smithi077.stdout:676.out 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:677 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:677.out 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:678 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:678.out 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:679 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:679.out 2023-07-04T18:53:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:680 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:680.out 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:681 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:681.out 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:682 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:682.out 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:683 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:683.out 2023-07-04T18:53:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:684 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:684.out 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:685 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:685.out 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:686 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:686.out 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:687 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:687.out 2023-07-04T18:53:44.367 INFO:teuthology.orchestra.run.smithi077.stdout:688 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:688.out 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:689 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:689.out 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:690 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:690.out 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:691 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:691.out 2023-07-04T18:53:44.368 INFO:teuthology.orchestra.run.smithi077.stdout:692 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:692.out 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:693 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:693.out 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:694 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:694.out 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:695 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:695.out 2023-07-04T18:53:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:696 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:696.out 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:697 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:697.out 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:698 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:698.out 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:699 2023-07-04T18:53:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:699.out 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:700 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:700.out 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:701 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:701.out 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:702 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:702.out 2023-07-04T18:53:44.371 INFO:teuthology.orchestra.run.smithi077.stdout:703 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:703.out 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:704 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:704.out 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:705 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:705.out 2023-07-04T18:53:44.372 INFO:teuthology.orchestra.run.smithi077.stdout:706 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:706.out 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:707 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:707.out 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:708 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:708.out 2023-07-04T18:53:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:709 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:709.out 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:710 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:710.out 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:711 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:711.out 2023-07-04T18:53:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:712 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:712.out 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:713 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:713.out 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:714 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:714.out 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:715 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:715.out 2023-07-04T18:53:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:716 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:716.out 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:717 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:717.out 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:718 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:718.out 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:719 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:719.out 2023-07-04T18:53:44.376 INFO:teuthology.orchestra.run.smithi077.stdout:720 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:720.out 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:721 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:721.out 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:722 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:722.out 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:723 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:723.out 2023-07-04T18:53:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:724 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:724.out 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:725 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:725.out 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:726 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:726.out 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:727 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:727.out 2023-07-04T18:53:44.378 INFO:teuthology.orchestra.run.smithi077.stdout:728 2023-07-04T18:53:44.379 INFO:teuthology.orchestra.run.smithi077.stdout:728.out 2023-07-04T18:53:44.379 INFO:teuthology.orchestra.run.smithi077.stdout:group.list 2023-07-04T18:53:44.379 INFO:teuthology.orchestra.run.smithi077.stdout:Makefile 2023-07-04T18:53:44.379 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/001) 2023-07-04T18:53:44.601 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:44.605 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:44.606 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:44.607 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:44.607 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:49.507 INFO:teuthology.orchestra.run.smithi077.stdout:generic/001 5s 2023-07-04T18:53:49.511 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/001 2023-07-04T18:53:49.512 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:53:49.513 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:50.229 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:53:50.229 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/002) 2023-07-04T18:53:50.430 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:50.434 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:50.435 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:50.435 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:50.436 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:51.033 INFO:teuthology.orchestra.run.smithi077.stdout:generic/002 1s 2023-07-04T18:53:51.037 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/002 2023-07-04T18:53:51.037 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:53:51.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:51.105 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:53:51.106 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/003) 2023-07-04T18:53:51.321 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:51.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:51.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:53:51.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:53:51.633 INFO:teuthology.orchestra.run.smithi077.stdout:generic/003 [not run] atime not maintained by ceph 2023-07-04T18:53:51.633 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/003 2023-07-04T18:53:51.633 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/003 2023-07-04T18:53:51.634 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:53:51.635 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:51.661 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:53:51.662 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/004) 2023-07-04T18:53:51.931 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:51.935 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:51.936 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:51.937 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:51.937 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:52.256 INFO:teuthology.orchestra.run.smithi077.stdout:generic/004 [not run] O_TMPFILE is not supported 2023-07-04T18:53:52.260 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/004 2023-07-04T18:53:52.260 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/004 2023-07-04T18:53:52.260 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:53:52.262 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:52.285 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:53:52.286 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/005) 2023-07-04T18:53:52.499 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:52.507 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:52.508 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:52.508 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:52.508 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:52.999 INFO:teuthology.orchestra.run.smithi077.stdout:generic/005 0s 2023-07-04T18:53:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/005 2023-07-04T18:53:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:53:53.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:53:53.121 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:53:53.122 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/006) 2023-07-04T18:53:53.324 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:53:53.328 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:53:53.329 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:53:53.329 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:53:53.329 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:54:18.467 INFO:teuthology.orchestra.run.smithi077.stdout:generic/006 25s 2023-07-04T18:54:18.471 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/006 2023-07-04T18:54:18.472 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:54:18.473 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:54:18.527 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:54:18.528 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/007) 2023-07-04T18:54:18.761 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:54:18.765 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:54:18.765 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:54:18.766 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:54:18.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:54:21.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:54:21.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:54:51.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:54:51.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:55:09.068 INFO:teuthology.orchestra.run.smithi077.stdout:generic/007 50s 2023-07-04T18:55:09.072 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/007 2023-07-04T18:55:09.073 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:09.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:09.147 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:09.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/008) 2023-07-04T18:55:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:09.362 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:09.363 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:09.364 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:09.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:09.825 INFO:teuthology.orchestra.run.smithi077.stdout:generic/008 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-07-04T18:55:09.829 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/008 2023-07-04T18:55:09.832 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/008 2023-07-04T18:55:09.832 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:09.835 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:09.862 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:09.863 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/009) 2023-07-04T18:55:10.058 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:10.063 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:10.063 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:10.064 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:10.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:10.452 INFO:teuthology.orchestra.run.smithi077.stdout:generic/009 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-07-04T18:55:10.456 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/009 2023-07-04T18:55:10.456 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/009 2023-07-04T18:55:10.457 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:10.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:10.483 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:10.484 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/010) 2023-07-04T18:55:10.683 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:10.688 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:10.688 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:10.689 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:10.689 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:11.011 INFO:teuthology.orchestra.run.smithi077.stdout:generic/010 [not run] /tmp/tmp.pA346HXQMhxfstests-dev/src/dbtest not built 2023-07-04T18:55:11.015 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/010 2023-07-04T18:55:11.015 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/010 2023-07-04T18:55:11.015 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:11.017 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:11.032 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:11.033 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/011) 2023-07-04T18:55:11.250 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:11.254 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:11.254 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:11.255 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:11.255 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:21.910 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:55:21.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:55:41.735 INFO:teuthology.orchestra.run.smithi077.stdout:generic/011 30s 2023-07-04T18:55:41.740 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/011 2023-07-04T18:55:41.741 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:41.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:41.948 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:41.949 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/012) 2023-07-04T18:55:42.176 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:42.180 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:42.180 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:42.181 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:42.182 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:42.596 INFO:teuthology.orchestra.run.smithi077.stdout:generic/012 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T18:55:42.600 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/012 2023-07-04T18:55:42.600 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/012 2023-07-04T18:55:42.600 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:55:42.602 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:42.657 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:55:42.657 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/013) 2023-07-04T18:55:42.863 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:55:42.867 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:55:42.868 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:55:42.868 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:55:42.869 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:55:51.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:55:51.992 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:56:22.417 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:56:22.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:56:31.411 INFO:teuthology.orchestra.run.smithi077.stdout:generic/013 48s 2023-07-04T18:56:31.482 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/013 2023-07-04T18:56:31.482 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T18:56:31.485 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:56:31.520 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T18:56:31.521 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/014) 2023-07-04T18:56:31.848 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T18:56:31.852 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T18:56:31.852 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T18:56:31.854 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T18:56:31.854 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T18:56:52.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:56:52.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:57:23.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:57:23.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:57:53.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:57:53.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:58:23.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:58:23.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:58:53.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:58:53.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:59:23.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:59:23.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:59:53.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T18:59:53.857 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:00:24.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:00:24.268 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:00:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:generic/014 260s 2023-07-04T19:00:52.977 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/014 2023-07-04T19:00:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:52.980 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:53.033 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:53.033 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/015) 2023-07-04T19:00:53.359 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:53.372 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:53.373 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:53.373 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:53.374 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:53.779 INFO:teuthology.orchestra.run.smithi077.stdout:generic/015 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:00:53.783 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/015 2023-07-04T19:00:53.784 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/015 2023-07-04T19:00:53.784 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:53.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:53.822 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:53.823 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/016) 2023-07-04T19:00:54.183 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:54.187 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:54.188 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:54.189 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:54.190 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:54.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:00:54.403 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:00:54.808 INFO:teuthology.orchestra.run.smithi077.stdout:generic/016 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:00:54.813 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/016 2023-07-04T19:00:54.813 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/016 2023-07-04T19:00:54.814 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:54.815 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:55.038 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:55.039 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/017) 2023-07-04T19:00:55.413 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:55.417 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:55.418 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:55.418 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:55.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:55.864 INFO:teuthology.orchestra.run.smithi077.stdout:generic/017 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T19:00:55.869 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/017 2023-07-04T19:00:55.869 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/017 2023-07-04T19:00:55.869 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:55.872 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:55.908 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:55.909 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/018) 2023-07-04T19:00:56.242 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:56.246 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:56.246 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:56.247 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:56.248 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:56.906 INFO:teuthology.orchestra.run.smithi077.stdout:generic/018 [not run] defragmentation not supported for fstype "ceph" 2023-07-04T19:00:56.923 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/018 2023-07-04T19:00:56.924 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/018 2023-07-04T19:00:56.924 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:56.925 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:56.956 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:56.957 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/019) 2023-07-04T19:00:57.282 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:57.287 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:57.287 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:57.288 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:57.288 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:57.741 INFO:teuthology.orchestra.run.smithi077.stdout:generic/019 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:00:57.749 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/019 2023-07-04T19:00:57.750 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/019 2023-07-04T19:00:57.750 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:00:57.752 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:00:57.778 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:00:57.779 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/020) 2023-07-04T19:00:58.082 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:00:58.087 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:00:58.088 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:00:58.089 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:00:58.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:01.217 INFO:teuthology.orchestra.run.smithi077.stdout:generic/020 3s 2023-07-04T19:01:01.222 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/020 2023-07-04T19:01:01.222 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:01.224 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:01.457 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:01.458 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/021) 2023-07-04T19:01:01.843 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:01:01.851 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:01:01.852 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:01:01.854 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:01:01.854 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:02.495 INFO:teuthology.orchestra.run.smithi077.stdout:generic/021 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:01:02.503 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/021 2023-07-04T19:01:02.503 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/021 2023-07-04T19:01:02.503 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:02.505 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:02.780 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:02.781 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/022) 2023-07-04T19:01:03.135 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:01:03.141 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:01:03.142 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:01:03.143 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:01:03.143 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:03.763 INFO:teuthology.orchestra.run.smithi077.stdout:generic/022 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:01:03.771 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/022 2023-07-04T19:01:03.771 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/022 2023-07-04T19:01:03.772 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:03.774 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:04.086 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:04.087 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/023) 2023-07-04T19:01:04.416 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:01:04.425 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:01:04.425 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:01:04.426 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:01:04.427 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:24.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:01:24.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:01:54.563 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:01:54.565 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:01:57.114 INFO:teuthology.orchestra.run.smithi077.stdout:generic/023 53s 2023-07-04T19:01:57.185 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/023 2023-07-04T19:01:57.186 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:57.188 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:57.472 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:57.472 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/024) 2023-07-04T19:01:57.891 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:01:57.897 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:01:57.898 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:01:57.899 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:01:57.899 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:58.369 INFO:teuthology.orchestra.run.smithi077.stdout:generic/024 [not run] kernel doesn't support renameat2 syscall 2023-07-04T19:01:58.373 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/024 2023-07-04T19:01:58.373 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/024 2023-07-04T19:01:58.373 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:58.375 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:58.636 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:58.637 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/025) 2023-07-04T19:01:59.038 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:01:59.052 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:01:59.053 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:01:59.054 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:01:59.054 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:59.543 INFO:teuthology.orchestra.run.smithi077.stdout:generic/025 [not run] kernel doesn't support renameat2 syscall 2023-07-04T19:01:59.547 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/025 2023-07-04T19:01:59.548 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/025 2023-07-04T19:01:59.548 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:01:59.550 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:01:59.793 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:01:59.794 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/026) 2023-07-04T19:02:00.203 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:00.208 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:00.208 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:00.209 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:00.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:00.750 INFO:teuthology.orchestra.run.smithi077.stdout:generic/026 [not run] ceph does not define maximum ACL count 2023-07-04T19:02:00.754 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/026 2023-07-04T19:02:00.755 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/026 2023-07-04T19:02:00.755 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:00.756 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:00.795 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:00.796 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/027) 2023-07-04T19:02:01.114 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:01.118 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:01.119 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:01.120 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:01.120 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:01.544 INFO:teuthology.orchestra.run.smithi077.stdout:generic/027 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:02:01.548 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/027 2023-07-04T19:02:01.548 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/027 2023-07-04T19:02:01.548 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:01.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:01.584 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:01.585 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/028) 2023-07-04T19:02:01.910 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:01.926 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:01.927 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:01.927 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:01.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:07.386 INFO:teuthology.orchestra.run.smithi077.stdout:generic/028 5s 2023-07-04T19:02:07.394 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/028 2023-07-04T19:02:07.395 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:07.397 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:07.426 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:07.427 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/029) 2023-07-04T19:02:07.759 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:07.763 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:07.763 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:07.767 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:07.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:12.374 INFO:teuthology.orchestra.run.smithi077.stdout:generic/029 4s 2023-07-04T19:02:12.382 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/029 2023-07-04T19:02:12.382 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:12.384 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:12.394 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:12.395 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/030) 2023-07-04T19:02:12.745 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:12.752 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:12.753 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:12.754 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:12.754 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:17.455 INFO:teuthology.orchestra.run.smithi077.stdout:generic/030 4s 2023-07-04T19:02:17.464 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/030 2023-07-04T19:02:17.464 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:17.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:17.475 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:17.476 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/031) 2023-07-04T19:02:17.807 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:17.812 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:17.813 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:17.813 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:17.814 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:18.369 INFO:teuthology.orchestra.run.smithi077.stdout:generic/031 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) 2023-07-04T19:02:18.374 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/031 2023-07-04T19:02:18.375 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/031 2023-07-04T19:02:18.375 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:18.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:18.411 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:18.412 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/032) 2023-07-04T19:02:18.743 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:18.751 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:18.751 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:18.752 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:18.753 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:19.228 INFO:teuthology.orchestra.run.smithi077.stdout:generic/032 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:19.232 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/032 2023-07-04T19:02:19.233 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/032 2023-07-04T19:02:19.233 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:19.235 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:19.276 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:19.277 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/033) 2023-07-04T19:02:19.590 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:19.597 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:19.599 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:19.600 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:19.600 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:20.171 INFO:teuthology.orchestra.run.smithi077.stdout:generic/033 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-07-04T19:02:20.176 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/033 2023-07-04T19:02:20.176 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/033 2023-07-04T19:02:20.177 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:20.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:20.216 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:20.217 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/034) 2023-07-04T19:02:20.557 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:20.562 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:20.562 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:20.563 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:20.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:21.057 INFO:teuthology.orchestra.run.smithi077.stdout:generic/034 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:21.072 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/034 2023-07-04T19:02:21.072 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/034 2023-07-04T19:02:21.072 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:21.075 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:21.104 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:21.105 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/035) 2023-07-04T19:02:21.419 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:21.424 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:21.425 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:21.426 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:21.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:22.058 INFO:teuthology.orchestra.run.smithi077.stdout:generic/035 1s 2023-07-04T19:02:22.063 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/035 2023-07-04T19:02:22.063 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:22.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:22.365 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:22.365 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/036) 2023-07-04T19:02:22.611 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:22.615 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:22.616 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:22.617 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:22.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:24.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:02:24.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:02:33.037 INFO:teuthology.orchestra.run.smithi077.stdout:generic/036 11s 2023-07-04T19:02:33.042 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/036 2023-07-04T19:02:33.042 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:33.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:33.227 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:33.228 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/037) 2023-07-04T19:02:33.543 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:33.547 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:33.548 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:33.548 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:33.549 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:36.084 INFO:teuthology.orchestra.run.smithi077.stdout:generic/037 3s 2023-07-04T19:02:36.088 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/037 2023-07-04T19:02:36.089 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:36.090 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:36.098 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:36.099 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/038) 2023-07-04T19:02:36.294 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:36.298 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:36.298 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:36.299 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:36.299 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:36.595 INFO:teuthology.orchestra.run.smithi077.stdout:generic/038 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:36.599 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/038 2023-07-04T19:02:36.599 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/038 2023-07-04T19:02:36.599 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:36.601 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:36.626 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:36.627 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/039) 2023-07-04T19:02:36.830 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:36.834 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:36.835 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:36.836 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:36.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:37.153 INFO:teuthology.orchestra.run.smithi077.stdout:generic/039 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:37.157 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/039 2023-07-04T19:02:37.157 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/039 2023-07-04T19:02:37.157 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:37.159 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:37.198 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:37.198 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/040) 2023-07-04T19:02:37.429 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:37.433 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:37.434 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:37.434 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:37.435 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:37.778 INFO:teuthology.orchestra.run.smithi077.stdout:generic/040 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:37.781 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/040 2023-07-04T19:02:37.782 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/040 2023-07-04T19:02:37.782 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:37.784 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:37.807 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:37.807 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/041) 2023-07-04T19:02:38.037 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:38.041 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:38.041 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:38.042 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:38.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:38.364 INFO:teuthology.orchestra.run.smithi077.stdout:generic/041 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:38.368 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/041 2023-07-04T19:02:38.369 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/041 2023-07-04T19:02:38.369 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:38.370 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:38.404 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:38.405 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/042) 2023-07-04T19:02:38.611 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:38.615 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:38.616 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:38.616 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:38.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:39.099 INFO:teuthology.orchestra.run.smithi077.stdout:generic/042 [not run] ceph does not support shutdown 2023-07-04T19:02:39.127 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/042 2023-07-04T19:02:39.128 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/042 2023-07-04T19:02:39.129 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:39.129 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:39.157 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:39.158 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/043) 2023-07-04T19:02:39.357 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:39.361 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:39.361 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:39.362 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:39.362 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:39.834 INFO:teuthology.orchestra.run.smithi077.stdout:generic/043 [not run] ceph does not support shutdown 2023-07-04T19:02:39.851 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/043 2023-07-04T19:02:39.851 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/043 2023-07-04T19:02:39.852 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:39.853 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:39.881 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:39.882 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/044) 2023-07-04T19:02:40.075 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:40.079 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:40.080 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:40.080 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:40.080 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:40.567 INFO:teuthology.orchestra.run.smithi077.stdout:generic/044 [not run] ceph does not support shutdown 2023-07-04T19:02:40.593 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/044 2023-07-04T19:02:40.593 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/044 2023-07-04T19:02:40.593 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:40.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:40.620 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:40.620 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/045) 2023-07-04T19:02:40.850 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:40.854 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:40.854 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:40.855 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:40.855 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:41.361 INFO:teuthology.orchestra.run.smithi077.stdout:generic/045 [not run] ceph does not support shutdown 2023-07-04T19:02:41.384 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/045 2023-07-04T19:02:41.385 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/045 2023-07-04T19:02:41.385 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:41.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:41.408 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:41.409 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/046) 2023-07-04T19:02:41.639 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:41.643 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:41.644 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:41.644 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:41.645 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:42.108 INFO:teuthology.orchestra.run.smithi077.stdout:generic/046 [not run] ceph does not support shutdown 2023-07-04T19:02:42.130 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/046 2023-07-04T19:02:42.130 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/046 2023-07-04T19:02:42.130 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:42.132 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:42.157 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:42.157 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/047) 2023-07-04T19:02:42.358 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:42.363 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:42.364 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:42.364 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:42.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:42.839 INFO:teuthology.orchestra.run.smithi077.stdout:generic/047 [not run] ceph does not support shutdown 2023-07-04T19:02:42.856 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/047 2023-07-04T19:02:42.856 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/047 2023-07-04T19:02:42.857 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:42.858 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:42.877 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:42.878 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/048) 2023-07-04T19:02:43.083 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:43.087 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:43.088 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:43.089 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:43.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:43.551 INFO:teuthology.orchestra.run.smithi077.stdout:generic/048 [not run] ceph does not support shutdown 2023-07-04T19:02:43.575 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/048 2023-07-04T19:02:43.575 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/048 2023-07-04T19:02:43.575 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:43.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:43.601 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:43.602 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/049) 2023-07-04T19:02:43.806 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:43.810 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:43.811 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:43.811 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:43.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:44.266 INFO:teuthology.orchestra.run.smithi077.stdout:generic/049 [not run] ceph does not support shutdown 2023-07-04T19:02:44.292 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/049 2023-07-04T19:02:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/049 2023-07-04T19:02:44.293 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:44.294 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:44.325 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:44.326 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/050) 2023-07-04T19:02:44.535 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:44.539 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:44.540 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:44.544 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:44.544 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:45.042 INFO:teuthology.orchestra.run.smithi077.stdout:generic/050 [not run] ceph does not support shutdown 2023-07-04T19:02:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/050 2023-07-04T19:02:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/050 2023-07-04T19:02:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:45.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:45.097 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:45.098 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/051) 2023-07-04T19:02:45.291 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:45.295 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:45.296 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:45.299 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:45.300 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:45.747 INFO:teuthology.orchestra.run.smithi077.stdout:generic/051 [not run] ceph does not support shutdown 2023-07-04T19:02:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/051 2023-07-04T19:02:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/051 2023-07-04T19:02:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:45.753 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:45.778 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:45.779 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/052) 2023-07-04T19:02:45.979 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:45.983 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:45.983 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:45.984 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:45.984 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:46.440 INFO:teuthology.orchestra.run.smithi077.stdout:generic/052 [not run] ceph does not support shutdown 2023-07-04T19:02:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/052 2023-07-04T19:02:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/052 2023-07-04T19:02:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:46.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:46.505 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:46.505 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/053) 2023-07-04T19:02:46.710 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:46.714 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:46.714 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:46.715 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:47.546 INFO:teuthology.orchestra.run.smithi077.stdout:generic/053 1s 2023-07-04T19:02:47.581 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/053 2023-07-04T19:02:47.581 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:47.583 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:47.602 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:47.603 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/054) 2023-07-04T19:02:47.811 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:47.814 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:47.815 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:47.816 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:47.816 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:48.316 INFO:teuthology.orchestra.run.smithi077.stdout:generic/054 [not run] ceph does not support shutdown 2023-07-04T19:02:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/054 2023-07-04T19:02:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/054 2023-07-04T19:02:48.339 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:48.340 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:48.359 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:48.360 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/055) 2023-07-04T19:02:48.564 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:48.568 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:48.569 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:48.570 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:48.570 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:49.064 INFO:teuthology.orchestra.run.smithi077.stdout:generic/055 [not run] ceph does not support shutdown 2023-07-04T19:02:49.094 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/055 2023-07-04T19:02:49.095 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/055 2023-07-04T19:02:49.095 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:49.096 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:49.118 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:49.118 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/056) 2023-07-04T19:02:49.322 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:49.327 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:49.327 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:49.327 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:49.327 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:49.661 INFO:teuthology.orchestra.run.smithi077.stdout:generic/056 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:49.665 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/056 2023-07-04T19:02:49.666 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/056 2023-07-04T19:02:49.666 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:49.667 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:49.693 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:49.694 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/057) 2023-07-04T19:02:49.894 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:49.899 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:49.899 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:49.900 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:49.900 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:50.226 INFO:teuthology.orchestra.run.smithi077.stdout:generic/057 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:50.230 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/057 2023-07-04T19:02:50.230 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/057 2023-07-04T19:02:50.230 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:50.232 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:50.260 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:50.260 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/058) 2023-07-04T19:02:50.454 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:50.459 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:50.459 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:50.460 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:50.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:50.864 INFO:teuthology.orchestra.run.smithi077.stdout:generic/058 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:50.868 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/058 2023-07-04T19:02:50.868 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/058 2023-07-04T19:02:50.869 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:50.870 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:51.188 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:51.189 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/059) 2023-07-04T19:02:51.426 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:51.430 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:51.431 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:51.431 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:51.431 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:51.755 INFO:teuthology.orchestra.run.smithi077.stdout:generic/059 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:51.759 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/059 2023-07-04T19:02:51.759 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/059 2023-07-04T19:02:51.760 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:51.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:51.794 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:51.794 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/060) 2023-07-04T19:02:52.002 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:52.006 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:52.007 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:52.007 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:52.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:52.412 INFO:teuthology.orchestra.run.smithi077.stdout:generic/060 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:52.416 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/060 2023-07-04T19:02:52.416 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/060 2023-07-04T19:02:52.416 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:52.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:52.690 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:52.691 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/061) 2023-07-04T19:02:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:52.940 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:52.942 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:53.327 INFO:teuthology.orchestra.run.smithi077.stdout:generic/061 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:53.331 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/061 2023-07-04T19:02:53.331 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/061 2023-07-04T19:02:53.332 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:53.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:53.449 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:53.450 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/062) 2023-07-04T19:02:53.853 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:53.857 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:53.857 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:53.858 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:53.859 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:54.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:02:54.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:02:55.688 INFO:teuthology.orchestra.run.smithi077.stdout:generic/062 1s 2023-07-04T19:02:55.699 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/062 2023-07-04T19:02:55.700 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:55.702 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:55.714 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:55.715 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/063) 2023-07-04T19:02:55.946 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:55.950 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:55.951 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:55.952 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:55.952 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:56.418 INFO:teuthology.orchestra.run.smithi077.stdout:generic/063 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:02:56.422 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/063 2023-07-04T19:02:56.422 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/063 2023-07-04T19:02:56.422 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:56.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:56.705 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:56.706 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/064) 2023-07-04T19:02:56.950 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:56.954 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:56.954 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:56.955 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:56.955 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:57.301 INFO:teuthology.orchestra.run.smithi077.stdout:generic/064 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T19:02:57.305 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/064 2023-07-04T19:02:57.305 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/064 2023-07-04T19:02:57.305 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:57.307 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:57.330 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:57.331 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/065) 2023-07-04T19:02:57.534 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:57.538 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:57.539 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:57.540 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:57.540 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:57.900 INFO:teuthology.orchestra.run.smithi077.stdout:generic/065 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:57.904 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/065 2023-07-04T19:02:57.904 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/065 2023-07-04T19:02:57.904 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:57.906 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:57.931 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:57.931 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/066) 2023-07-04T19:02:58.136 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:58.141 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:58.141 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:58.142 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:58.142 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:58.449 INFO:teuthology.orchestra.run.smithi077.stdout:generic/066 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:58.453 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/066 2023-07-04T19:02:58.453 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/066 2023-07-04T19:02:58.453 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:58.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:58.476 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:58.477 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/067) 2023-07-04T19:02:58.710 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:58.714 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:58.714 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:58.715 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:58.715 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:59.108 INFO:teuthology.orchestra.run.smithi077.stdout:generic/067 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:02:59.112 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/067 2023-07-04T19:02:59.112 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/067 2023-07-04T19:02:59.113 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:59.114 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:59.147 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:59.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/068) 2023-07-04T19:02:59.341 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:59.346 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:59.347 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:59.347 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:59.347 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:59.673 INFO:teuthology.orchestra.run.smithi077.stdout:generic/068 [not run] ceph does not support freezing 2023-07-04T19:02:59.676 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/068 2023-07-04T19:02:59.677 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/068 2023-07-04T19:02:59.677 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:02:59.679 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:02:59.706 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:02:59.706 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/069) 2023-07-04T19:02:59.901 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:02:59.906 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:02:59.906 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:02:59.907 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:02:59.907 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:05.465 INFO:teuthology.orchestra.run.smithi077.stdout:generic/069 5s 2023-07-04T19:03:05.480 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/069 2023-07-04T19:03:05.480 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:05.482 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:05.496 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:05.496 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/070) 2023-07-04T19:03:05.703 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:05.707 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:05.707 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:05.708 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:05.708 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:24.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:03:24.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:03:28.068 INFO:teuthology.orchestra.run.smithi077.stdout:generic/070 23s 2023-07-04T19:03:28.073 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/070 2023-07-04T19:03:28.073 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:28.075 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:28.435 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:28.435 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/071) 2023-07-04T19:03:28.761 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:28.765 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:28.765 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:28.768 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:28.768 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:29.220 INFO:teuthology.orchestra.run.smithi077.stdout:generic/071 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T19:03:29.226 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/071 2023-07-04T19:03:29.226 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/071 2023-07-04T19:03:29.226 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:29.228 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:29.255 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:29.256 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/072) 2023-07-04T19:03:29.555 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:29.563 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:29.564 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:29.564 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:29.565 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:30.112 INFO:teuthology.orchestra.run.smithi077.stdout:generic/072 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) 2023-07-04T19:03:30.116 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/072 2023-07-04T19:03:30.116 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/072 2023-07-04T19:03:30.116 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:30.118 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:30.158 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:30.158 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/073) 2023-07-04T19:03:30.485 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:30.493 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:30.493 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:30.494 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:30.495 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:30.926 INFO:teuthology.orchestra.run.smithi077.stdout:generic/073 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:03:30.931 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/073 2023-07-04T19:03:30.931 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/073 2023-07-04T19:03:30.932 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:30.933 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:30.971 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:30.972 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/074) 2023-07-04T19:03:31.303 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:31.310 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:31.310 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:31.314 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:49.773 INFO:teuthology.orchestra.run.smithi077.stdout:generic/074 18s 2023-07-04T19:03:49.786 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/074 2023-07-04T19:03:49.787 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:03:49.791 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:50.356 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:03:50.357 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/075) 2023-07-04T19:03:50.723 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:03:50.733 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:03:50.733 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:03:50.734 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:03:50.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:03:54.999 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:03:55.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:03:55.063 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T19:03:55.062+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140135) UID: 0 2023-07-04T19:03:55.064 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T19:03:55.063+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140135) UID: 0 2023-07-04T19:04:26.368 INFO:teuthology.orchestra.run.smithi077.stdout:generic/075 35s 2023-07-04T19:04:26.373 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/075 2023-07-04T19:04:26.373 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:26.375 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:26.718 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:26.719 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/076) 2023-07-04T19:04:27.064 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:27.068 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:27.069 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:27.069 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:27.070 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:27.520 INFO:teuthology.orchestra.run.smithi077.stdout:generic/076 [not run] require 172.21.15.77:6789:/scratch to be local device 2023-07-04T19:04:27.525 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/076 2023-07-04T19:04:27.525 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/076 2023-07-04T19:04:27.526 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:27.527 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:27.555 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:27.556 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/077) 2023-07-04T19:04:27.837 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:27.845 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:27.845 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:27.846 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:27.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:30.879 INFO:teuthology.orchestra.run.smithi077.stdout:generic/077 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:04:30.883 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/077 2023-07-04T19:04:30.884 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/077 2023-07-04T19:04:30.884 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:30.886 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:31.153 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:31.153 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/078) 2023-07-04T19:04:31.537 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:31.542 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:31.542 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:31.543 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:31.543 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:32.032 INFO:teuthology.orchestra.run.smithi077.stdout:generic/078 [not run] kernel doesn't support renameat2 syscall 2023-07-04T19:04:32.039 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/078 2023-07-04T19:04:32.039 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/078 2023-07-04T19:04:32.039 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:32.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:32.283 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:32.284 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/079) 2023-07-04T19:04:32.677 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:32.680 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:32.682 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:32.683 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:32.683 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:33.172 INFO:teuthology.orchestra.run.smithi077.stdout:generic/079 [not run] file system doesn't support chattr +ia 2023-07-04T19:04:33.176 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/079 2023-07-04T19:04:33.177 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/079 2023-07-04T19:04:33.177 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:33.179 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:33.212 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:33.212 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/080) 2023-07-04T19:04:33.496 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:33.506 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:33.506 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:33.507 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:33.508 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:36.067 INFO:teuthology.orchestra.run.smithi077.stdout:generic/080 3s 2023-07-04T19:04:36.073 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/080 2023-07-04T19:04:36.073 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:36.075 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:36.121 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:36.121 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/081) 2023-07-04T19:04:36.421 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:36.425 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:36.425 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:36.430 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:36.430 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:36.896 INFO:teuthology.orchestra.run.smithi077.stdout:generic/081 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:04:36.905 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/081 2023-07-04T19:04:36.905 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/081 2023-07-04T19:04:36.905 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:36.907 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:36.937 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:36.937 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/082) 2023-07-04T19:04:37.251 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:37.259 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:37.260 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:37.261 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:37.261 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:37.723 INFO:teuthology.orchestra.run.smithi077.stdout:generic/082 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:04:37.728 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/082 2023-07-04T19:04:37.728 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/082 2023-07-04T19:04:37.729 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:37.730 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:37.759 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:37.760 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/083) 2023-07-04T19:04:38.092 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:38.097 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:38.097 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:38.098 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:38.099 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:38.550 INFO:teuthology.orchestra.run.smithi077.stdout:generic/083 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:04:38.555 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/083 2023-07-04T19:04:38.556 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/083 2023-07-04T19:04:38.556 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:38.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:38.585 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:38.586 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/084) 2023-07-04T19:04:38.876 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:38.880 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:38.880 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:38.881 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:38.882 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:44.945 INFO:teuthology.orchestra.run.smithi077.stdout:generic/084 5s 2023-07-04T19:04:44.954 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/084 2023-07-04T19:04:44.954 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:44.957 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:44.976 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:44.976 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/085) 2023-07-04T19:04:45.269 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:45.278 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:45.279 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:45.279 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:45.280 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:45.845 INFO:teuthology.orchestra.run.smithi077.stdout:generic/085 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:04:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/085 2023-07-04T19:04:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/085 2023-07-04T19:04:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:45.851 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:45.884 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:45.884 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/086) 2023-07-04T19:04:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:46.191 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:46.192 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:46.192 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:46.193 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:46.598 INFO:teuthology.orchestra.run.smithi077.stdout:generic/086 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:04:46.602 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/086 2023-07-04T19:04:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/086 2023-07-04T19:04:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:46.605 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:46.640 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:46.641 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/087) 2023-07-04T19:04:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:46.957 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:46.957 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:47.570 INFO:teuthology.orchestra.run.smithi077.stdout:generic/087 0s 2023-07-04T19:04:47.575 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/087 2023-07-04T19:04:47.576 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:47.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:47.922 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:47.923 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/088) 2023-07-04T19:04:48.265 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:48.270 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:48.270 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:48.272 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:48.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:48.781 INFO:teuthology.orchestra.run.smithi077.stdout:generic/088 0s 2023-07-04T19:04:48.868 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/088 2023-07-04T19:04:48.868 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:04:48.870 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:04:49.169 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:04:49.170 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/089) 2023-07-04T19:04:49.455 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:04:49.464 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:04:49.464 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:04:49.465 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:04:49.465 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:06:58.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:06:58.265 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:07:29.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:07:29.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:07:59.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:07:59.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:08:30.422 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:08:30.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:09:00.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:09:00.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:09:00.889 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T19:09:00.887+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140286) UID: 0 2023-07-04T19:09:00.890 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T19:09:00.887+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140286) UID: 0 2023-07-04T19:11:57.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:11:57.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:12:27.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:12:27.210 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:12:57.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:12:57.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:12:58.123 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T19:12:58.120+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140423) UID: 0 2023-07-04T19:12:58.123 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T19:12:58.121+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 140423) UID: 0 2023-07-04T19:13:23.456 INFO:teuthology.orchestra.run.smithi077.stdout:generic/089 514s 2023-07-04T19:13:23.465 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/089 2023-07-04T19:13:23.465 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:13:23.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:13:23.898 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:13:23.899 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/090) 2023-07-04T19:13:24.187 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:13:24.190 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:13:24.191 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:13:24.192 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:13:24.192 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:13:24.643 INFO:teuthology.orchestra.run.smithi077.stdout:generic/090 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:13:24.647 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/090 2023-07-04T19:13:24.648 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/090 2023-07-04T19:13:24.648 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:13:24.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:13:24.677 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:13:24.677 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/091) 2023-07-04T19:13:24.883 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:13:24.887 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:13:24.888 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:13:24.888 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:13:24.889 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:05.162 INFO:teuthology.orchestra.run.smithi077.stdout:generic/091 100s 2023-07-04T19:15:05.167 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/091 2023-07-04T19:15:05.167 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:05.169 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:05.328 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:05.328 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/092) 2023-07-04T19:15:05.733 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:05.737 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:05.737 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:05.738 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:05.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:06.078 INFO:teuthology.orchestra.run.smithi077.stdout:generic/092 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:15:06.082 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/092 2023-07-04T19:15:06.082 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/092 2023-07-04T19:15:06.082 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:06.084 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:06.106 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:06.107 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/093) 2023-07-04T19:15:06.307 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:06.311 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:06.312 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:06.312 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:06.313 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:06.822 INFO:teuthology.orchestra.run.smithi077.stdout:generic/093 0s 2023-07-04T19:15:06.827 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/093 2023-07-04T19:15:06.827 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:06.829 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:07.167 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:07.168 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/094) 2023-07-04T19:15:07.396 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:07.400 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:07.401 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:07.401 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:07.402 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:07.748 INFO:teuthology.orchestra.run.smithi077.stdout:generic/094 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T19:15:07.752 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/094 2023-07-04T19:15:07.753 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/094 2023-07-04T19:15:07.753 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:07.754 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:08.036 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:08.037 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/095) 2023-07-04T19:15:08.298 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:08.302 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:08.302 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:08.303 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:08.303 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:21.373 INFO:teuthology.orchestra.run.smithi077.stdout:generic/095 13s 2023-07-04T19:15:21.377 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/095 2023-07-04T19:15:21.378 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:21.379 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:21.387 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:21.388 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/096) 2023-07-04T19:15:21.589 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:21.593 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:21.594 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:21.595 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:21.595 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:21.983 INFO:teuthology.orchestra.run.smithi077.stdout:generic/096 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-07-04T19:15:21.987 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/096 2023-07-04T19:15:21.988 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/096 2023-07-04T19:15:21.989 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:21.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:22.012 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:22.013 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/097) 2023-07-04T19:15:22.215 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:22.218 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:22.219 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:22.220 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:22.220 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:23.005 INFO:teuthology.orchestra.run.smithi077.stdout:generic/097 1s 2023-07-04T19:15:23.011 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/097 2023-07-04T19:15:23.011 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:23.013 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:23.040 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:23.040 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/098) 2023-07-04T19:15:23.254 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:23.259 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:23.259 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:23.260 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:23.260 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:25.141 INFO:teuthology.orchestra.run.smithi077.stdout:generic/098 2s 2023-07-04T19:15:25.146 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/098 2023-07-04T19:15:25.146 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:25.148 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:25.156 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:25.157 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/099) 2023-07-04T19:15:25.387 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:25.390 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:25.391 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:25.392 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:25.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:26.326 INFO:teuthology.orchestra.run.smithi077.stdout:generic/099 1s 2023-07-04T19:15:26.330 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/099 2023-07-04T19:15:26.331 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:26.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:26.601 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:26.602 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/100) 2023-07-04T19:15:26.916 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:26.920 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:26.921 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:26.922 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:26.922 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:39.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:15:39.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:15:41.391 INFO:teuthology.orchestra.run.smithi077.stdout:generic/100 14s 2023-07-04T19:15:41.397 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/100 2023-07-04T19:15:41.397 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:41.399 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:42.515 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:42.515 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/101) 2023-07-04T19:15:42.911 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:42.915 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:42.915 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:42.916 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:42.916 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:43.257 INFO:teuthology.orchestra.run.smithi077.stdout:generic/101 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:15:43.261 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/101 2023-07-04T19:15:43.261 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/101 2023-07-04T19:15:43.261 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:43.263 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:43.282 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:43.283 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/102) 2023-07-04T19:15:43.478 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:43.482 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:43.483 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:43.484 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:43.484 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:43.798 INFO:teuthology.orchestra.run.smithi077.stdout:generic/102 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:15:43.802 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/102 2023-07-04T19:15:43.802 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/102 2023-07-04T19:15:43.803 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:43.804 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:43.827 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:43.828 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/103) 2023-07-04T19:15:44.022 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:44.026 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:44.027 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:44.027 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:44.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:44.366 INFO:teuthology.orchestra.run.smithi077.stdout:generic/103 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:15:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/103 2023-07-04T19:15:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/103 2023-07-04T19:15:44.370 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:44.372 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:44.678 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:44.679 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/104) 2023-07-04T19:15:44.929 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:44.933 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:44.934 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:44.935 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:44.935 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:45.276 INFO:teuthology.orchestra.run.smithi077.stdout:generic/104 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:15:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/104 2023-07-04T19:15:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/104 2023-07-04T19:15:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:45.282 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:45.313 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:45.314 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/105) 2023-07-04T19:15:45.515 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:45.519 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:45.519 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:45.520 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:45.520 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:46.406 INFO:teuthology.orchestra.run.smithi077.stdout:generic/105 1s 2023-07-04T19:15:46.415 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/105 2023-07-04T19:15:46.416 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:46.423 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:46.432 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:46.433 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/106) 2023-07-04T19:15:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:46.645 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:46.646 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:46.646 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:46.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:47.022 INFO:teuthology.orchestra.run.smithi077.stdout:generic/106 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:15:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/106 2023-07-04T19:15:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/106 2023-07-04T19:15:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:47.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:47.058 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:47.059 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/107) 2023-07-04T19:15:47.269 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:47.273 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:47.274 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:47.275 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:47.275 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:47.638 INFO:teuthology.orchestra.run.smithi077.stdout:generic/107 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:15:47.642 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/107 2023-07-04T19:15:47.642 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/107 2023-07-04T19:15:47.642 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:47.644 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:47.673 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:47.673 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/108) 2023-07-04T19:15:47.902 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:47.906 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:47.907 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:47.908 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:47.908 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:48.355 INFO:teuthology.orchestra.run.smithi077.stdout:generic/108 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T19:15:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/108 2023-07-04T19:15:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/108 2023-07-04T19:15:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:48.361 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:48.389 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:48.390 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/109) 2023-07-04T19:15:48.586 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:15:48.590 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:15:48.590 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:15:48.591 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:15:48.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:59.099 INFO:teuthology.orchestra.run.smithi077.stdout:generic/109 11s 2023-07-04T19:15:59.112 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/109 2023-07-04T19:15:59.112 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:15:59.115 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:15:59.139 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:15:59.140 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/110) 2023-07-04T19:16:00.042 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:00.046 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:00.047 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:00.048 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:00.048 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:10.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:16:10.156 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:16:21.191 INFO:teuthology.orchestra.run.smithi077.stdout:generic/110 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:16:21.195 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/110 2023-07-04T19:16:21.195 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/110 2023-07-04T19:16:21.195 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:21.197 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:21.486 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:21.487 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/111) 2023-07-04T19:16:22.084 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:22.088 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:22.088 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:22.089 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:22.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:22.453 INFO:teuthology.orchestra.run.smithi077.stdout:generic/111 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:16:22.456 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/111 2023-07-04T19:16:22.457 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/111 2023-07-04T19:16:22.457 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:22.458 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:22.779 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:22.779 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/112) 2023-07-04T19:16:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:23.053 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:23.054 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:23.055 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:23.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:39.582 INFO:teuthology.orchestra.run.smithi077.stdout:generic/112 16s 2023-07-04T19:16:39.586 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/112 2023-07-04T19:16:39.587 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:39.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:39.952 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:39.952 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/113) 2023-07-04T19:16:40.159 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:40.162 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:40.163 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:40.164 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:40.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:40.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:16:40.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:16:54.325 INFO:teuthology.orchestra.run.smithi077.stdout:generic/113 14s 2023-07-04T19:16:54.329 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/113 2023-07-04T19:16:54.329 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:54.331 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:54.676 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:54.677 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/114) 2023-07-04T19:16:55.045 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:55.049 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:55.049 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:55.050 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:55.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:55.414 INFO:teuthology.orchestra.run.smithi077.stdout:generic/114 [not run] device block size: 4096 greater than 512 2023-07-04T19:16:55.418 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/114 2023-07-04T19:16:55.419 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/114 2023-07-04T19:16:55.419 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:55.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:55.450 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:55.451 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/115) 2023-07-04T19:16:55.648 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:55.652 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:55.653 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:55.654 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:55.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:56.010 INFO:teuthology.orchestra.run.smithi077.stdout:generic/115 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:16:56.014 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/115 2023-07-04T19:16:56.015 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/115 2023-07-04T19:16:56.015 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:56.016 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:56.308 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:56.309 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/116) 2023-07-04T19:16:56.606 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:56.609 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:56.610 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:56.611 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:56.611 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:56.974 INFO:teuthology.orchestra.run.smithi077.stdout:generic/116 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:16:56.977 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/116 2023-07-04T19:16:56.977 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/116 2023-07-04T19:16:56.978 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:16:56.979 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:16:57.319 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:16:57.320 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/117) 2023-07-04T19:16:57.541 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:16:57.545 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:16:57.545 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:16:57.546 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:16:57.546 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:08.250 INFO:teuthology.orchestra.run.smithi077.stdout:generic/117 11s 2023-07-04T19:17:08.255 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/117 2023-07-04T19:17:08.256 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:08.257 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:08.266 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:08.266 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/118) 2023-07-04T19:17:08.473 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:08.478 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:08.478 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:08.479 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:08.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:09.244 INFO:teuthology.orchestra.run.smithi077.stdout:generic/118 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:17:09.248 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/118 2023-07-04T19:17:09.249 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/118 2023-07-04T19:17:09.249 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:09.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:09.600 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:09.601 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/119) 2023-07-04T19:17:09.918 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:09.922 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:09.922 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:09.923 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:09.923 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:10.264 INFO:teuthology.orchestra.run.smithi077.stdout:generic/119 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:17:10.268 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/119 2023-07-04T19:17:10.268 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/119 2023-07-04T19:17:10.269 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:10.270 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:10.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:17:10.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:17:10.632 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:10.633 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/120) 2023-07-04T19:17:10.865 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:10.869 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:10.870 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:10.870 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:10.871 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:11.203 INFO:teuthology.orchestra.run.smithi077.stdout:generic/120 [not run] atime not maintained by ceph 2023-07-04T19:17:11.207 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/120 2023-07-04T19:17:11.207 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/120 2023-07-04T19:17:11.207 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:11.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:11.233 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:11.234 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/121) 2023-07-04T19:17:11.454 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:11.459 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:11.459 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:11.460 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:11.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:11.817 INFO:teuthology.orchestra.run.smithi077.stdout:generic/121 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T19:17:11.821 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/121 2023-07-04T19:17:11.822 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/121 2023-07-04T19:17:11.822 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:11.823 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:12.059 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:12.059 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/122) 2023-07-04T19:17:12.376 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:12.380 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:12.381 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:12.381 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:12.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:12.750 INFO:teuthology.orchestra.run.smithi077.stdout:generic/122 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T19:17:12.754 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/122 2023-07-04T19:17:12.754 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/122 2023-07-04T19:17:12.754 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:12.756 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:13.097 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:13.098 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/123) 2023-07-04T19:17:13.321 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:13.324 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:13.325 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:13.325 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:13.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:13.810 INFO:teuthology.orchestra.run.smithi077.stdout:generic/123 0s 2023-07-04T19:17:13.815 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/123 2023-07-04T19:17:13.815 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:13.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:14.144 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:14.145 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/124) 2023-07-04T19:17:14.384 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:14.388 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:14.389 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:14.389 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:14.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:16.616 INFO:teuthology.orchestra.run.smithi077.stdout:generic/124 2s 2023-07-04T19:17:16.634 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/124 2023-07-04T19:17:16.634 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:17:16.637 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:17.105 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:17:17.106 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/125) 2023-07-04T19:17:17.490 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:17:17.494 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:17:17.495 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:17:17.495 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:17:17.495 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:17:40.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:17:40.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:18:10.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:18:10.790 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:18:19.006 INFO:teuthology.orchestra.run.smithi077.stdout:generic/125 62s 2023-07-04T19:18:19.011 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/125 2023-07-04T19:18:19.012 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:18:19.014 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:18:19.369 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:18:19.370 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/126) 2023-07-04T19:18:19.604 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:18:19.608 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:18:19.609 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:18:19.609 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:18:19.610 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:18:20.058 INFO:teuthology.orchestra.run.smithi077.stdout:generic/126 1s 2023-07-04T19:18:20.063 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/126 2023-07-04T19:18:20.063 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:18:20.065 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:18:20.452 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:18:20.452 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/127) 2023-07-04T19:18:20.675 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:18:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:18:20.679 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:18:20.680 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:18:20.680 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:18:40.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:18:40.862 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:19:10.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:19:10.954 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:19:41.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:19:41.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:20:11.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:20:11.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:20:41.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:20:41.629 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:21:11.693 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:21:11.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:21:42.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:21:42.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:22:12.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:22:12.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:22:42.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:22:42.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:23:12.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:23:12.495 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:23:42.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:23:42.575 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:24:12.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:24:12.641 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:24:42.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:24:42.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:25:13.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:25:13.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:25:43.074 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:25:43.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:26:13.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:26:13.150 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:26:43.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:26:43.429 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:27:13.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:27:13.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:27:31.811 INFO:teuthology.orchestra.run.smithi077.stdout:generic/127 551s 2023-07-04T19:27:31.816 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/127 2023-07-04T19:27:31.816 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:27:31.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:27:32.174 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:27:32.175 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/128) 2023-07-04T19:27:32.390 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:27:32.394 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:27:32.394 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:27:32.395 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:27:32.395 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:27:33.420 INFO:teuthology.orchestra.run.smithi077.stdout:generic/128 1s 2023-07-04T19:27:33.424 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/128 2023-07-04T19:27:33.425 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:27:33.429 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:27:33.438 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:27:33.438 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/129) 2023-07-04T19:27:33.642 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:27:33.646 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:27:33.647 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:27:33.648 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:27:33.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:27:43.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:27:43.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:28:13.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:28:13.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:28:43.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:28:43.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:29:14.012 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:29:14.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:29:44.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:29:44.171 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:30:14.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:30:14.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:30:44.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:30:44.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:31:14.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:31:14.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:31:44.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:31:45.234 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:32:15.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:32:15.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:32:45.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:32:45.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:33:15.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:33:15.815 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:33:46.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:33:46.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:34:16.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:34:16.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:34:46.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:34:46.357 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:35:16.545 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:35:16.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:35:46.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:35:46.733 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:36:17.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:36:17.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:36:47.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:36:47.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:37:17.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:37:17.678 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:37:47.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:37:47.936 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:38:18.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:38:18.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:38:48.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:38:48.331 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:39:18.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:39:18.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:39:49.092 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:39:49.173 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:39:49.302 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T19:39:49.299+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 142174) UID: 0 2023-07-04T19:39:49.305 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T19:39:49.304+0000 7fb432fbc700 -1 Fail to open '/proc/142174/cmdline' error = (2) No such file or directory 2023-07-04T19:39:49.306 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T19:39:49.304+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 142174) UID: 0 2023-07-04T19:39:49.309 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T19:39:49.300+0000 7f218b0dd700 -1 Fail to open '/proc/142174/cmdline' error = (2) No such file or directory 2023-07-04T19:39:49.309 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T19:39:49.300+0000 7f218b0dd700 -1 received signal: Hangup from (PID: 142174) UID: 0 2023-07-04T19:39:49.325 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T19:39:49.299+0000 7efd0b7fe700 -1 Fail to open '/proc/142174/cmdline' error = (2) No such file or directory 2023-07-04T19:39:49.325 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T19:39:49.299+0000 7efd0b7fe700 -1 received signal: Hangup from (PID: 142174) UID: 0 2023-07-04T19:44:01.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:44:01.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:44:01.328 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T19:44:01.326+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 255169) UID: 0 2023-07-04T19:44:01.328 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T19:44:01.326+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 255169) UID: 0 2023-07-04T19:44:01.328 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T19:44:01.326+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 255169) UID: 0 2023-07-04T19:44:01.335 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T19:44:01.329+0000 7f2bac08a700 -1 Fail to open '/proc/255169/cmdline' error = (2) No such file or directory 2023-07-04T19:44:01.335 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T19:44:01.329+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 255169) UID: 0 2023-07-04T19:47:21.329 INFO:teuthology.orchestra.run.smithi077.stdout:generic/129 1188s 2023-07-04T19:47:21.370 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/129 2023-07-04T19:47:21.370 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:47:21.384 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:21.392 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:47:21.393 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/130) 2023-07-04T19:47:21.874 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:47:21.878 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:47:21.878 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:47:21.879 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:47:21.879 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:35.033 INFO:teuthology.orchestra.run.smithi077.stdout:generic/130 13s 2023-07-04T19:47:35.042 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/130 2023-07-04T19:47:35.042 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:47:35.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:35.053 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:47:35.054 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/131) 2023-07-04T19:47:35.270 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:47:35.274 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:47:35.276 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:47:35.276 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:47:35.277 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:36.978 INFO:teuthology.orchestra.run.smithi077.stdout:generic/131 1s 2023-07-04T19:47:36.993 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/131 2023-07-04T19:47:36.993 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:47:36.995 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:37.177 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:47:37.179 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/132) 2023-07-04T19:47:37.749 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:47:37.753 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:47:37.753 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:47:37.754 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:47:37.755 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:51.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:47:51.364 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:47:51.406 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T19:47:51.405+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 142341) UID: 0 2023-07-04T19:47:51.407 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T19:47:51.405+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 142341) UID: 0 2023-07-04T19:47:51.407 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T19:47:51.405+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 142341) UID: 0 2023-07-04T19:47:51.407 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T19:47:51.405+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 142341) UID: 0 2023-07-04T19:47:51.591 INFO:teuthology.orchestra.run.smithi077.stdout:generic/132 13s 2023-07-04T19:47:51.596 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/132 2023-07-04T19:47:51.596 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:47:51.598 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:47:51.606 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:47:51.606 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/133) 2023-07-04T19:47:51.808 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:47:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:47:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:47:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:47:51.813 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:50:22.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:50:22.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:50:52.506 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:50:52.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:50:52.552 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T19:50:52.551+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 258190) UID: 0 2023-07-04T19:50:52.553 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T19:50:52.551+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 258190) UID: 0 2023-07-04T19:50:52.553 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T19:50:52.551+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 258190) UID: 0 2023-07-04T19:50:52.554 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T19:50:52.551+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 258190) UID: 0 2023-07-04T19:51:47.078 INFO:teuthology.orchestra.run.smithi077.stdout:generic/133 235s 2023-07-04T19:51:47.083 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/133 2023-07-04T19:51:47.083 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:47.085 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:47.337 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:47.337 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/134) 2023-07-04T19:51:47.916 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:47.920 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:47.920 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:47.921 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:48.294 INFO:teuthology.orchestra.run.smithi077.stdout:generic/134 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/134 2023-07-04T19:51:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/134 2023-07-04T19:51:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:48.300 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:48.665 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:48.666 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/135) 2023-07-04T19:51:48.906 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:48.910 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:48.910 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:48.911 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:48.911 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:50.044 INFO:teuthology.orchestra.run.smithi077.stdout:generic/135 1s 2023-07-04T19:51:50.049 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/135 2023-07-04T19:51:50.049 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:50.051 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:50.059 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:50.060 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/136) 2023-07-04T19:51:50.266 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:50.270 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:50.271 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:50.271 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:50.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:50.634 INFO:teuthology.orchestra.run.smithi077.stdout:generic/136 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T19:51:50.638 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/136 2023-07-04T19:51:50.638 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/136 2023-07-04T19:51:50.639 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:50.640 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:51.035 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:51.036 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/137) 2023-07-04T19:51:51.273 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:51.277 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:51.278 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:51.279 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:51.279 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:51.654 INFO:teuthology.orchestra.run.smithi077.stdout:generic/137 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:51.658 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/137 2023-07-04T19:51:51.658 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/137 2023-07-04T19:51:51.658 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:51.660 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:52.025 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:52.025 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/138) 2023-07-04T19:51:52.310 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:52.314 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:52.314 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:52.315 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:52.315 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:52.671 INFO:teuthology.orchestra.run.smithi077.stdout:generic/138 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/138 2023-07-04T19:51:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/138 2023-07-04T19:51:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:52.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:53.025 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:53.025 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/139) 2023-07-04T19:51:53.281 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:53.285 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:53.286 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:53.287 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:53.287 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:53.649 INFO:teuthology.orchestra.run.smithi077.stdout:generic/139 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/139 2023-07-04T19:51:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/139 2023-07-04T19:51:53.653 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:53.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:53.986 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:53.987 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/140) 2023-07-04T19:51:54.288 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:54.292 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:54.293 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:54.294 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:54.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:54.641 INFO:teuthology.orchestra.run.smithi077.stdout:generic/140 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:54.645 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/140 2023-07-04T19:51:54.645 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/140 2023-07-04T19:51:54.645 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:54.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:55.052 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:55.052 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/141) 2023-07-04T19:51:55.256 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:55.260 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:55.261 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:55.261 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:55.261 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:55.812 INFO:teuthology.orchestra.run.smithi077.stdout:generic/141 0s 2023-07-04T19:51:55.817 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/141 2023-07-04T19:51:55.817 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:55.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:55.826 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:55.827 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/142) 2023-07-04T19:51:56.054 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:56.058 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:56.059 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:56.059 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:56.059 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:56.424 INFO:teuthology.orchestra.run.smithi077.stdout:generic/142 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:56.428 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/142 2023-07-04T19:51:56.428 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/142 2023-07-04T19:51:56.428 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:56.430 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:56.759 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:56.760 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/143) 2023-07-04T19:51:57.117 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:57.121 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:57.122 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:57.122 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:57.122 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:57.475 INFO:teuthology.orchestra.run.smithi077.stdout:generic/143 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:57.485 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/143 2023-07-04T19:51:57.485 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/143 2023-07-04T19:51:57.486 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:57.486 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:57.878 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:57.879 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/144) 2023-07-04T19:51:58.087 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:58.091 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:58.092 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:58.093 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:58.093 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:58.443 INFO:teuthology.orchestra.run.smithi077.stdout:generic/144 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:58.447 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/144 2023-07-04T19:51:58.447 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/144 2023-07-04T19:51:58.447 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:58.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:58.707 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:58.708 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/145) 2023-07-04T19:51:59.066 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:51:59.070 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:51:59.071 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:51:59.071 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:51:59.072 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:59.436 INFO:teuthology.orchestra.run.smithi077.stdout:generic/145 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:51:59.439 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/145 2023-07-04T19:51:59.440 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/145 2023-07-04T19:51:59.440 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:51:59.442 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:51:59.838 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:51:59.839 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/146) 2023-07-04T19:52:00.043 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:00.047 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:00.048 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:00.048 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:00.049 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:00.412 INFO:teuthology.orchestra.run.smithi077.stdout:generic/146 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:00.416 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/146 2023-07-04T19:52:00.417 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/146 2023-07-04T19:52:00.417 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:00.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:00.781 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:00.784 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/147) 2023-07-04T19:52:01.048 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:01.051 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:01.052 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:01.053 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:01.053 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:01.431 INFO:teuthology.orchestra.run.smithi077.stdout:generic/147 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:01.434 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/147 2023-07-04T19:52:01.434 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/147 2023-07-04T19:52:01.435 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:01.436 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:01.488 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:01.488 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/148) 2023-07-04T19:52:02.036 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:02.039 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:02.040 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:02.040 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:02.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:02.392 INFO:teuthology.orchestra.run.smithi077.stdout:generic/148 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:02.396 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/148 2023-07-04T19:52:02.396 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/148 2023-07-04T19:52:02.397 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:02.398 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:02.781 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:02.781 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/149) 2023-07-04T19:52:03.028 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:03.032 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:03.033 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:03.033 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:03.033 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:03.400 INFO:teuthology.orchestra.run.smithi077.stdout:generic/149 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:03.404 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/149 2023-07-04T19:52:03.404 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/149 2023-07-04T19:52:03.404 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:03.406 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:03.815 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:03.816 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/150) 2023-07-04T19:52:04.041 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:04.045 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:04.046 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:04.046 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:04.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:04.433 INFO:teuthology.orchestra.run.smithi077.stdout:generic/150 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:04.437 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/150 2023-07-04T19:52:04.438 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/150 2023-07-04T19:52:04.438 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:04.439 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:04.707 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:04.708 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/151) 2023-07-04T19:52:05.035 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:05.039 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:05.040 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:05.041 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:05.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:05.400 INFO:teuthology.orchestra.run.smithi077.stdout:generic/151 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:05.404 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/151 2023-07-04T19:52:05.405 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/151 2023-07-04T19:52:05.405 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:05.406 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:05.769 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:05.770 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/152) 2023-07-04T19:52:06.022 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:06.026 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:06.026 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:06.027 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:06.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:06.414 INFO:teuthology.orchestra.run.smithi077.stdout:generic/152 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:06.418 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/152 2023-07-04T19:52:06.418 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/152 2023-07-04T19:52:06.418 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:06.420 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:06.797 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:06.798 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/153) 2023-07-04T19:52:07.037 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:07.041 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:07.042 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:07.042 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:07.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:07.381 INFO:teuthology.orchestra.run.smithi077.stdout:generic/153 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:07.385 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/153 2023-07-04T19:52:07.385 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/153 2023-07-04T19:52:07.386 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:07.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:07.782 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:07.783 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/154) 2023-07-04T19:52:07.985 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:07.989 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:07.990 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:07.990 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:07.990 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:08.359 INFO:teuthology.orchestra.run.smithi077.stdout:generic/154 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:08.362 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/154 2023-07-04T19:52:08.363 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/154 2023-07-04T19:52:08.363 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:08.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:08.759 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:08.759 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/155) 2023-07-04T19:52:09.022 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:09.026 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:09.026 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:09.027 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:09.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:09.410 INFO:teuthology.orchestra.run.smithi077.stdout:generic/155 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:09.414 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/155 2023-07-04T19:52:09.414 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/155 2023-07-04T19:52:09.414 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:09.416 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:09.816 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:09.817 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/156) 2023-07-04T19:52:10.038 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:10.042 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:10.042 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:10.043 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:10.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:10.409 INFO:teuthology.orchestra.run.smithi077.stdout:generic/156 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:10.413 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/156 2023-07-04T19:52:10.413 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/156 2023-07-04T19:52:10.413 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:10.415 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:10.813 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:10.813 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/157) 2023-07-04T19:52:11.025 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:11.029 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:11.030 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:11.030 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:11.030 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:11.394 INFO:teuthology.orchestra.run.smithi077.stdout:generic/157 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:11.398 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/157 2023-07-04T19:52:11.398 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/157 2023-07-04T19:52:11.398 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:11.400 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:11.799 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:11.799 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/158) 2023-07-04T19:52:12.052 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:12.056 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:12.056 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:12.057 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:12.057 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:12.412 INFO:teuthology.orchestra.run.smithi077.stdout:generic/158 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T19:52:12.416 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/158 2023-07-04T19:52:12.417 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/158 2023-07-04T19:52:12.417 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:12.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:12.789 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:12.789 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/159) 2023-07-04T19:52:13.035 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:13.039 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:13.040 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:13.040 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:13.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:13.413 INFO:teuthology.orchestra.run.smithi077.stdout:generic/159 [not run] file system doesn't support chattr +i 2023-07-04T19:52:13.417 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/159 2023-07-04T19:52:13.417 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/159 2023-07-04T19:52:13.418 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:13.419 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:13.452 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:13.453 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/160) 2023-07-04T19:52:13.652 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:13.656 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:13.657 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:13.657 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:13.658 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:14.008 INFO:teuthology.orchestra.run.smithi077.stdout:generic/160 [not run] file system doesn't support chattr +i 2023-07-04T19:52:14.012 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/160 2023-07-04T19:52:14.012 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/160 2023-07-04T19:52:14.013 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:14.014 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:14.045 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:14.046 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/161) 2023-07-04T19:52:14.252 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:14.256 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:14.257 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:14.257 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:14.258 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:14.874 INFO:teuthology.orchestra.run.smithi077.stdout:generic/161 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:15.144 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/161 2023-07-04T19:52:15.144 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/161 2023-07-04T19:52:15.144 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:15.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:15.231 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:15.232 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/162) 2023-07-04T19:52:15.433 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:15.436 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:15.437 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:15.438 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:15.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:15.961 INFO:teuthology.orchestra.run.smithi077.stdout:generic/162 [not run] Dedupe not supported by scratch filesystem type: ceph 2023-07-04T19:52:16.244 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/162 2023-07-04T19:52:16.245 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/162 2023-07-04T19:52:16.245 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:16.325 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:16.384 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:16.384 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/163) 2023-07-04T19:52:16.591 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:16.594 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:16.595 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:16.596 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:16.596 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:17.145 INFO:teuthology.orchestra.run.smithi077.stdout:generic/163 [not run] Dedupe not supported by scratch filesystem type: ceph 2023-07-04T19:52:17.503 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/163 2023-07-04T19:52:17.503 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/163 2023-07-04T19:52:17.503 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:17.506 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:17.572 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:17.572 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/164) 2023-07-04T19:52:17.793 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:17.797 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:17.798 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:17.799 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:17.799 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:18.690 INFO:teuthology.orchestra.run.smithi077.stdout:generic/164 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:18.703 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/164 2023-07-04T19:52:18.703 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/164 2023-07-04T19:52:18.703 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:18.704 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:18.736 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:18.736 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/165) 2023-07-04T19:52:18.940 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:18.943 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:18.944 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:18.945 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:18.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:19.644 INFO:teuthology.orchestra.run.smithi077.stdout:generic/165 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:19.867 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/165 2023-07-04T19:52:19.867 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/165 2023-07-04T19:52:19.867 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:19.874 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:19.894 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:19.895 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/166) 2023-07-04T19:52:20.101 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:20.105 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:20.106 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:20.106 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:20.107 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:20.851 INFO:teuthology.orchestra.run.smithi077.stdout:generic/166 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:20.858 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/166 2023-07-04T19:52:20.858 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/166 2023-07-04T19:52:20.858 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:20.867 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:21.046 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:21.047 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/167) 2023-07-04T19:52:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:21.260 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:21.261 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:21.262 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:21.262 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:22.133 INFO:teuthology.orchestra.run.smithi077.stdout:generic/167 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:22.156 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/167 2023-07-04T19:52:22.157 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/167 2023-07-04T19:52:22.157 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:22.174 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:22.207 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:22.208 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/168) 2023-07-04T19:52:22.411 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:22.415 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:22.415 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:22.416 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:22.416 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:23.282 INFO:teuthology.orchestra.run.smithi077.stdout:generic/168 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:23.296 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/168 2023-07-04T19:52:23.296 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/168 2023-07-04T19:52:23.296 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:23.315 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:23.368 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:23.369 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/169) 2023-07-04T19:52:23.571 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:23.575 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:23.576 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:23.576 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:23.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:24.356 INFO:teuthology.orchestra.run.smithi077.stdout:generic/169 1s 2023-07-04T19:52:24.361 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/169 2023-07-04T19:52:24.361 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:24.363 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:24.371 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:24.372 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/170) 2023-07-04T19:52:24.599 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:24.603 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:24.604 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:24.605 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:24.605 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:25.409 INFO:teuthology.orchestra.run.smithi077.stdout:generic/170 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:25.478 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/170 2023-07-04T19:52:25.478 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/170 2023-07-04T19:52:25.479 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:25.506 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:25.538 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:25.539 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/171) 2023-07-04T19:52:25.739 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:25.743 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:25.744 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:25.745 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:25.745 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:26.589 INFO:teuthology.orchestra.run.smithi077.stdout:generic/171 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:26.603 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/171 2023-07-04T19:52:26.603 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/171 2023-07-04T19:52:26.604 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:26.612 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:26.719 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:26.720 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/172) 2023-07-04T19:52:26.936 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:26.940 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:26.941 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:26.941 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:26.942 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:27.775 INFO:teuthology.orchestra.run.smithi077.stdout:generic/172 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:27.796 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/172 2023-07-04T19:52:27.804 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/172 2023-07-04T19:52:27.804 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:27.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:27.865 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:27.866 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/173) 2023-07-04T19:52:28.073 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:28.077 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:28.077 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:28.078 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:28.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:28.977 INFO:teuthology.orchestra.run.smithi077.stdout:generic/173 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:28.998 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/173 2023-07-04T19:52:28.998 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/173 2023-07-04T19:52:28.998 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:29.001 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:29.028 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:29.029 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/174) 2023-07-04T19:52:29.244 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:29.248 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:29.249 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:29.250 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:29.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:30.075 INFO:teuthology.orchestra.run.smithi077.stdout:generic/174 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:30.159 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/174 2023-07-04T19:52:30.160 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/174 2023-07-04T19:52:30.160 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:30.169 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:30.195 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:30.196 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/175) 2023-07-04T19:52:30.400 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:30.404 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:30.405 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:30.405 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:30.406 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:31.226 INFO:teuthology.orchestra.run.smithi077.stdout:generic/175 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:31.268 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/175 2023-07-04T19:52:31.268 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/175 2023-07-04T19:52:31.269 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:31.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:31.351 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:31.352 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/176) 2023-07-04T19:52:31.572 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:31.576 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:31.576 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:31.577 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:31.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:32.534 INFO:teuthology.orchestra.run.smithi077.stdout:generic/176 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:32.539 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/176 2023-07-04T19:52:32.539 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/176 2023-07-04T19:52:32.539 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:32.541 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:32.565 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:32.565 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/177) 2023-07-04T19:52:32.780 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:32.784 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:32.785 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:32.786 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:32.786 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:33.268 INFO:teuthology.orchestra.run.smithi077.stdout:generic/177 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T19:52:33.272 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/177 2023-07-04T19:52:33.272 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/177 2023-07-04T19:52:33.272 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:33.274 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:33.604 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:33.605 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/178) 2023-07-04T19:52:33.900 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:33.904 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:33.904 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:33.905 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:33.905 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:34.287 INFO:teuthology.orchestra.run.smithi077.stdout:generic/178 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:34.291 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/178 2023-07-04T19:52:34.291 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/178 2023-07-04T19:52:34.291 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:34.293 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:34.525 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:34.526 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/179) 2023-07-04T19:52:34.942 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:34.946 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:34.946 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:34.947 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:34.947 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:35.332 INFO:teuthology.orchestra.run.smithi077.stdout:generic/179 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:35.336 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/179 2023-07-04T19:52:35.336 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/179 2023-07-04T19:52:35.336 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:35.338 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:35.682 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:35.682 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/180) 2023-07-04T19:52:35.963 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:35.968 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:35.968 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:35.969 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:35.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:36.333 INFO:teuthology.orchestra.run.smithi077.stdout:generic/180 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:36.336 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/180 2023-07-04T19:52:36.336 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/180 2023-07-04T19:52:36.337 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:36.338 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:36.733 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:36.734 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/181) 2023-07-04T19:52:36.972 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:36.976 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:36.977 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:36.977 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:36.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:37.346 INFO:teuthology.orchestra.run.smithi077.stdout:generic/181 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T19:52:37.350 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/181 2023-07-04T19:52:37.351 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/181 2023-07-04T19:52:37.351 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:37.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:37.740 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:37.741 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/182) 2023-07-04T19:52:37.966 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:37.971 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:37.971 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:37.972 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:37.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:38.323 INFO:teuthology.orchestra.run.smithi077.stdout:generic/182 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T19:52:38.326 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/182 2023-07-04T19:52:38.327 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/182 2023-07-04T19:52:38.327 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:38.328 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:38.744 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:38.744 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/183) 2023-07-04T19:52:38.978 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:38.982 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:38.983 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:38.984 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:38.984 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:39.676 INFO:teuthology.orchestra.run.smithi077.stdout:generic/183 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:39.857 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/183 2023-07-04T19:52:39.858 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/183 2023-07-04T19:52:39.858 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:39.876 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:39.932 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:39.933 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/184) 2023-07-04T19:52:40.148 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:40.152 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:40.153 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:40.154 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:40.154 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:40.526 INFO:teuthology.orchestra.run.smithi077.stdout:generic/184 0s 2023-07-04T19:52:40.531 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/184 2023-07-04T19:52:40.531 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:40.533 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:40.555 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:40.555 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/185) 2023-07-04T19:52:40.759 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:40.762 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:40.763 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:40.764 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:40.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:41.413 INFO:teuthology.orchestra.run.smithi077.stdout:generic/185 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:41.586 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/185 2023-07-04T19:52:41.586 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/185 2023-07-04T19:52:41.587 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:41.623 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:41.705 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:41.706 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/186) 2023-07-04T19:52:41.922 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:41.926 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:41.926 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:41.927 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:41.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:42.838 INFO:teuthology.orchestra.run.smithi077.stdout:generic/186 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:42.908 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/186 2023-07-04T19:52:42.908 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/186 2023-07-04T19:52:42.908 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:42.911 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:42.950 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:42.950 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/187) 2023-07-04T19:52:43.180 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:43.184 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:43.185 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:43.185 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:43.843 INFO:teuthology.orchestra.run.smithi077.stdout:generic/187 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:43.947 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/187 2023-07-04T19:52:43.948 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/187 2023-07-04T19:52:43.948 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:43.981 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:44.156 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:44.156 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/188) 2023-07-04T19:52:44.369 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:44.373 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:44.374 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:44.375 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:generic/188 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:45.070 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/188 2023-07-04T19:52:45.070 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/188 2023-07-04T19:52:45.071 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:45.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:45.256 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:45.257 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/189) 2023-07-04T19:52:45.551 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:45.555 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:45.556 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:45.557 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:45.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:generic/189 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/189 2023-07-04T19:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/189 2023-07-04T19:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:46.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:46.502 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:46.503 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/190) 2023-07-04T19:52:46.707 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:46.710 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:46.711 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:46.712 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:47.653 INFO:teuthology.orchestra.run.smithi077.stdout:generic/190 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/190 2023-07-04T19:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/190 2023-07-04T19:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:47.660 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:47.681 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:47.681 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/191) 2023-07-04T19:52:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:47.887 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:47.887 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:48.666 INFO:teuthology.orchestra.run.smithi077.stdout:generic/191 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:48.671 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/191 2023-07-04T19:52:48.672 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/191 2023-07-04T19:52:48.672 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:48.798 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:48.841 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:48.841 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/192) 2023-07-04T19:52:49.052 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:49.057 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:49.057 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:49.394 INFO:teuthology.orchestra.run.smithi077.stdout:generic/192 [not run] atime not maintained by ceph 2023-07-04T19:52:49.398 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/192 2023-07-04T19:52:49.399 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/192 2023-07-04T19:52:49.399 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:49.400 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:49.423 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:49.424 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/193) 2023-07-04T19:52:49.628 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:49.632 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:49.633 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:49.633 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:49.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:50.526 INFO:teuthology.orchestra.run.smithi077.stdout:generic/193 1s 2023-07-04T19:52:50.530 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/193 2023-07-04T19:52:50.531 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:50.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:50.745 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:50.746 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/194) 2023-07-04T19:52:51.155 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:51.159 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:51.159 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:51.160 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:51.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:51.987 INFO:teuthology.orchestra.run.smithi077.stdout:generic/194 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:52.055 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/194 2023-07-04T19:52:52.055 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/194 2023-07-04T19:52:52.056 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:52.064 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:52.102 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:52.103 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/195) 2023-07-04T19:52:52.303 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:52.307 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:52.308 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:52.309 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:52.309 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:53.204 INFO:teuthology.orchestra.run.smithi077.stdout:generic/195 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:53.248 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/195 2023-07-04T19:52:53.248 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/195 2023-07-04T19:52:53.248 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:53.267 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:53.291 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:53.292 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/196) 2023-07-04T19:52:53.519 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:53.523 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:53.523 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:53.524 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:53.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:54.304 INFO:teuthology.orchestra.run.smithi077.stdout:generic/196 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:54.405 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/196 2023-07-04T19:52:54.405 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/196 2023-07-04T19:52:54.406 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:54.430 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:54.493 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:54.493 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/197) 2023-07-04T19:52:54.701 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:54.705 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:54.706 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:54.707 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:54.707 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:55.628 INFO:teuthology.orchestra.run.smithi077.stdout:generic/197 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:55.634 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/197 2023-07-04T19:52:55.634 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/197 2023-07-04T19:52:55.634 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:55.641 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:55.671 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:55.672 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/198) 2023-07-04T19:52:55.872 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:55.876 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:55.876 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:55.877 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:55.877 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:56.607 INFO:teuthology.orchestra.run.smithi077.stdout:generic/198 0s 2023-07-04T19:52:56.612 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/198 2023-07-04T19:52:56.612 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:56.614 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:56.999 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:56.999 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/199) 2023-07-04T19:52:57.230 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:57.236 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:58.130 INFO:teuthology.orchestra.run.smithi077.stdout:generic/199 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:58.203 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/199 2023-07-04T19:52:58.203 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/199 2023-07-04T19:52:58.203 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:58.206 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:58.241 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:58.242 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/200) 2023-07-04T19:52:58.462 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:58.467 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:58.467 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:58.468 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:58.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:59.273 INFO:teuthology.orchestra.run.smithi077.stdout:generic/200 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:52:59.329 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/200 2023-07-04T19:52:59.329 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/200 2023-07-04T19:52:59.330 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:52:59.346 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:52:59.405 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:52:59.405 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/201) 2023-07-04T19:52:59.648 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:52:59.652 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:52:59.653 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:52:59.653 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:52:59.654 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:00.529 INFO:teuthology.orchestra.run.smithi077.stdout:generic/201 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:53:00.594 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/201 2023-07-04T19:53:00.594 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/201 2023-07-04T19:53:00.594 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:00.596 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:00.632 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:00.634 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/202) 2023-07-04T19:53:00.858 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:00.862 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:00.862 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:00.863 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:00.864 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:01.767 INFO:teuthology.orchestra.run.smithi077.stdout:generic/202 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:53:01.819 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/202 2023-07-04T19:53:01.819 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/202 2023-07-04T19:53:01.819 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:01.824 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:01.854 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:01.855 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/203) 2023-07-04T19:53:02.078 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:02.082 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:02.083 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:02.083 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:02.084 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:02.891 INFO:teuthology.orchestra.run.smithi077.stdout:generic/203 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:53:02.933 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/203 2023-07-04T19:53:02.933 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/203 2023-07-04T19:53:02.933 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:02.981 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:03.015 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:03.016 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/204) 2023-07-04T19:53:03.219 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:03.223 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:03.224 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:03.224 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:03.225 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:05.127 INFO:teuthology.orchestra.run.smithi077.stdout:generic/204 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:53:05.132 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/204 2023-07-04T19:53:05.132 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/204 2023-07-04T19:53:05.133 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:05.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:05.163 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:05.164 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/205) 2023-07-04T19:53:05.383 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:05.387 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:05.388 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:05.389 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:05.389 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:06.033 INFO:teuthology.orchestra.run.smithi077.stdout:generic/205 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:53:06.073 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/205 2023-07-04T19:53:06.074 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/205 2023-07-04T19:53:06.074 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:06.091 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:06.305 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:06.306 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/206) 2023-07-04T19:53:06.535 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:06.540 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:06.540 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:06.541 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:06.541 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:07.366 INFO:teuthology.orchestra.run.smithi077.stdout:generic/206 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:53:07.485 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/206 2023-07-04T19:53:07.485 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/206 2023-07-04T19:53:07.485 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:07.524 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:07.525 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/207) 2023-07-04T19:53:07.727 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:07.731 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:07.732 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:07.732 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:13.868 INFO:teuthology.orchestra.run.smithi077.stdout:generic/207 6s 2023-07-04T19:53:13.873 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/207 2023-07-04T19:53:13.873 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:53:13.875 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:14.141 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:53:14.142 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/208) 2023-07-04T19:53:14.457 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:53:14.461 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:53:14.462 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:53:14.463 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:53:14.463 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:53:34.814 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:53:34.817 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:54:04.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:54:04.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:54:34.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:54:34.972 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:55:05.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:55:05.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:55:35.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:55:35.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:56:05.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:56:05.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:56:34.886 INFO:teuthology.orchestra.run.smithi077.stdout:generic/208 200s 2023-07-04T19:56:34.891 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/208 2023-07-04T19:56:34.891 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:56:34.893 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:56:35.146 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:56:35.147 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/209) 2023-07-04T19:56:35.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:56:35.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:56:35.472 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:56:35.476 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:56:35.477 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:56:35.477 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:56:35.478 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:05.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:57:05.442 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:57:12.373 INFO:teuthology.orchestra.run.smithi077.stdout:generic/209 37s 2023-07-04T19:57:12.378 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/209 2023-07-04T19:57:12.378 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:12.696 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:12.697 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/210) 2023-07-04T19:57:12.951 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:12.955 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:12.955 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:12.956 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:12.956 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:13.419 INFO:teuthology.orchestra.run.smithi077.stdout:generic/210 0s 2023-07-04T19:57:13.424 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/210 2023-07-04T19:57:13.424 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:13.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:13.745 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:13.746 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/211) 2023-07-04T19:57:14.019 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:14.023 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:14.023 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:14.024 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:14.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:14.519 INFO:teuthology.orchestra.run.smithi077.stdout:generic/211 0s 2023-07-04T19:57:14.524 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/211 2023-07-04T19:57:14.524 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:14.526 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:14.558 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:14.559 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/212) 2023-07-04T19:57:14.761 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:14.766 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:14.766 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:14.767 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:14.767 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:15.199 INFO:teuthology.orchestra.run.smithi077.stdout:generic/212 1s 2023-07-04T19:57:15.206 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/212 2023-07-04T19:57:15.206 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:15.206 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:15.244 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:15.245 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/213) 2023-07-04T19:57:15.477 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:15.480 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:15.481 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:15.482 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:15.482 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:15.830 INFO:teuthology.orchestra.run.smithi077.stdout:generic/213 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:57:15.833 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/213 2023-07-04T19:57:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/213 2023-07-04T19:57:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:15.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:15.861 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:15.861 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/214) 2023-07-04T19:57:16.094 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:16.098 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:16.098 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:16.099 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:16.099 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:16.461 INFO:teuthology.orchestra.run.smithi077.stdout:generic/214 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/214 2023-07-04T19:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/214 2023-07-04T19:57:16.465 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:16.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:16.493 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:16.494 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/215) 2023-07-04T19:57:16.696 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:16.700 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:16.701 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:16.702 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:16.702 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:19.103 INFO:teuthology.orchestra.run.smithi077.stdout:generic/215 3s 2023-07-04T19:57:19.108 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/215 2023-07-04T19:57:19.109 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:19.111 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:19.157 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:19.157 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/216) 2023-07-04T19:57:19.395 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:19.399 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:19.400 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:19.400 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:19.401 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:20.258 INFO:teuthology.orchestra.run.smithi077.stdout:generic/216 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:20.272 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/216 2023-07-04T19:57:20.272 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/216 2023-07-04T19:57:20.273 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:20.280 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:20.326 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:20.326 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/217) 2023-07-04T19:57:20.532 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:20.536 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:20.537 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:20.537 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:20.538 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:21.401 INFO:teuthology.orchestra.run.smithi077.stdout:generic/217 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:21.448 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/217 2023-07-04T19:57:21.448 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/217 2023-07-04T19:57:21.449 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:21.451 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:21.490 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:21.490 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/218) 2023-07-04T19:57:21.695 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:21.699 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:21.699 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:21.700 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:21.701 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:22.512 INFO:teuthology.orchestra.run.smithi077.stdout:generic/218 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:22.534 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/218 2023-07-04T19:57:22.534 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/218 2023-07-04T19:57:22.534 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:22.568 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:22.607 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:22.609 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/219) 2023-07-04T19:57:22.812 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:22.817 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:22.817 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:22.818 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:22.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:23.146 INFO:teuthology.orchestra.run.smithi077.stdout:generic/219 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:23.150 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/219 2023-07-04T19:57:23.151 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/219 2023-07-04T19:57:23.151 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:23.153 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:23.178 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:23.179 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/220) 2023-07-04T19:57:23.385 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:23.389 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:23.390 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:23.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:24.236 INFO:teuthology.orchestra.run.smithi077.stdout:generic/220 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:24.302 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/220 2023-07-04T19:57:24.302 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/220 2023-07-04T19:57:24.303 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:24.305 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:24.351 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:24.351 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/221) 2023-07-04T19:57:24.573 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:24.577 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:24.578 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:24.578 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:24.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:25.936 INFO:teuthology.orchestra.run.smithi077.stdout:generic/221 1s 2023-07-04T19:57:25.942 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/221 2023-07-04T19:57:25.943 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:25.944 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:25.977 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:25.978 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/222) 2023-07-04T19:57:26.217 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:26.221 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:26.222 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:26.223 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:26.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:27.068 INFO:teuthology.orchestra.run.smithi077.stdout:generic/222 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:27.074 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/222 2023-07-04T19:57:27.075 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/222 2023-07-04T19:57:27.075 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:27.090 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:27.154 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:27.155 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/223) 2023-07-04T19:57:27.358 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:27.362 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:27.363 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:27.364 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:27.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:27.715 INFO:teuthology.orchestra.run.smithi077.stdout:generic/223 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:57:27.720 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/223 2023-07-04T19:57:27.720 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/223 2023-07-04T19:57:27.720 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:27.722 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:27.747 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:27.748 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/224) 2023-07-04T19:57:27.954 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:27.958 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:27.958 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:27.959 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:27.959 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:28.292 INFO:teuthology.orchestra.run.smithi077.stdout:generic/224 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:57:28.296 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/224 2023-07-04T19:57:28.297 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/224 2023-07-04T19:57:28.297 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:28.298 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:28.318 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:28.319 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/225) 2023-07-04T19:57:28.528 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:28.532 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:28.533 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:28.533 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:28.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:28.921 INFO:teuthology.orchestra.run.smithi077.stdout:generic/225 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T19:57:28.925 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/225 2023-07-04T19:57:28.925 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/225 2023-07-04T19:57:28.925 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:28.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:29.283 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:29.284 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/226) 2023-07-04T19:57:29.513 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:29.517 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:29.517 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:29.518 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:29.518 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:29.880 INFO:teuthology.orchestra.run.smithi077.stdout:generic/226 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T19:57:29.884 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/226 2023-07-04T19:57:29.885 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/226 2023-07-04T19:57:29.885 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:29.886 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:29.913 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:29.913 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/227) 2023-07-04T19:57:30.116 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:30.120 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:30.121 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:30.121 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:30.121 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:30.931 INFO:teuthology.orchestra.run.smithi077.stdout:generic/227 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:30.951 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/227 2023-07-04T19:57:30.951 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/227 2023-07-04T19:57:30.951 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:30.985 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:31.051 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:31.052 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/228) 2023-07-04T19:57:31.255 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:31.259 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:31.259 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:31.260 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:31.260 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:31.648 INFO:teuthology.orchestra.run.smithi077.stdout:generic/228 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T19:57:31.652 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/228 2023-07-04T19:57:31.653 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/228 2023-07-04T19:57:31.653 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:31.655 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:31.676 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:31.677 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/229) 2023-07-04T19:57:31.888 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:31.892 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:31.893 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:31.893 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:31.893 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:32.743 INFO:teuthology.orchestra.run.smithi077.stdout:generic/229 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:32.756 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/229 2023-07-04T19:57:32.757 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/229 2023-07-04T19:57:32.757 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:32.769 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:32.831 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:32.831 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/230) 2023-07-04T19:57:33.050 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:33.054 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:33.054 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:33.055 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:33.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:33.398 INFO:teuthology.orchestra.run.smithi077.stdout:generic/230 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:33.402 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/230 2023-07-04T19:57:33.402 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/230 2023-07-04T19:57:33.403 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:33.404 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:33.429 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:33.430 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/231) 2023-07-04T19:57:33.644 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:33.648 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:33.649 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:33.649 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:33.650 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:33.975 INFO:teuthology.orchestra.run.smithi077.stdout:generic/231 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:33.978 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/231 2023-07-04T19:57:33.979 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/231 2023-07-04T19:57:33.979 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:33.981 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:34.018 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:34.019 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/232) 2023-07-04T19:57:34.252 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:34.256 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:34.257 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:34.258 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:34.258 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:34.598 INFO:teuthology.orchestra.run.smithi077.stdout:generic/232 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:34.602 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/232 2023-07-04T19:57:34.602 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/232 2023-07-04T19:57:34.602 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:34.604 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:34.627 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:34.628 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/233) 2023-07-04T19:57:34.834 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:34.838 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:34.838 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:34.839 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:34.839 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:35.177 INFO:teuthology.orchestra.run.smithi077.stdout:generic/233 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:35.181 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/233 2023-07-04T19:57:35.181 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/233 2023-07-04T19:57:35.181 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:35.183 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:35.204 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:35.205 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/234) 2023-07-04T19:57:35.405 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:35.410 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:35.410 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:35.411 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:35.411 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:35.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:57:35.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:57:35.741 INFO:teuthology.orchestra.run.smithi077.stdout:generic/234 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:35.746 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/234 2023-07-04T19:57:35.746 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/234 2023-07-04T19:57:35.746 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:35.773 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:35.774 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/235) 2023-07-04T19:57:36.001 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:36.005 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:36.005 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:36.006 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:36.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:36.352 INFO:teuthology.orchestra.run.smithi077.stdout:generic/235 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T19:57:36.356 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/235 2023-07-04T19:57:36.356 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/235 2023-07-04T19:57:36.356 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:36.358 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:36.381 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:36.381 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/236) 2023-07-04T19:57:36.586 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:36.590 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:36.591 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:36.591 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:36.592 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:37.983 INFO:teuthology.orchestra.run.smithi077.stdout:generic/236 1s 2023-07-04T19:57:37.988 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/236 2023-07-04T19:57:37.989 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:37.991 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:38.366 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:38.367 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/237) 2023-07-04T19:57:38.579 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:38.583 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:38.584 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:38.585 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:38.585 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:39.001 INFO:teuthology.orchestra.run.smithi077.stdout:generic/237 0s 2023-07-04T19:57:39.006 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/237 2023-07-04T19:57:39.006 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:39.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:39.324 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:39.325 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/238) 2023-07-04T19:57:39.595 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:39.599 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:39.600 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:39.601 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:39.601 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:40.408 INFO:teuthology.orchestra.run.smithi077.stdout:generic/238 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T19:57:40.470 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/238 2023-07-04T19:57:40.471 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/238 2023-07-04T19:57:40.471 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:57:40.473 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:57:40.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:57:40.512 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/239) 2023-07-04T19:57:40.739 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:57:40.743 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:57:40.744 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:57:40.745 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:57:40.745 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:58:05.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:58:05.575 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T19:58:05.713 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T19:58:05.712+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 143083) UID: 0 2023-07-04T19:58:05.714 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T19:58:05.712+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 143083) UID: 0 2023-07-04T19:58:05.715 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T19:58:05.712+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 143083) UID: 0 2023-07-04T19:58:05.727 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T19:58:05.712+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 143083) UID: 0 2023-07-04T19:59:37.260 INFO:teuthology.orchestra.run.smithi077.stdout:generic/239 117s 2023-07-04T19:59:37.263 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/239 2023-07-04T19:59:37.264 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:59:37.265 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:59:37.625 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:59:37.625 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/240) 2023-07-04T19:59:37.855 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:59:37.859 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:59:37.860 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:59:37.860 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:59:37.861 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:59:38.543 INFO:teuthology.orchestra.run.smithi077.stdout:generic/240 0s 2023-07-04T19:59:38.548 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/240 2023-07-04T19:59:38.548 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T19:59:38.550 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T19:59:38.877 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T19:59:38.878 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/241) 2023-07-04T19:59:39.154 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T19:59:39.159 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T19:59:39.160 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T19:59:39.160 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T19:59:39.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:52.627 INFO:teuthology.orchestra.run.smithi077.stdout:generic/241 73s 2023-07-04T20:00:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/241 2023-07-04T20:00:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:00:52.654 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:55.463 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:00:55.464 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/242) 2023-07-04T20:00:56.274 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:00:56.277 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:00:56.278 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:00:56.279 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:00:56.279 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:57.618 INFO:teuthology.orchestra.run.smithi077.stdout:generic/242 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:00:57.668 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/242 2023-07-04T20:00:57.668 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/242 2023-07-04T20:00:57.669 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:00:57.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:57.764 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:00:57.764 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/243) 2023-07-04T20:00:58.018 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:00:58.022 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:00:58.022 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:00:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:00:58.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:58.821 INFO:teuthology.orchestra.run.smithi077.stdout:generic/243 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:00:58.878 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/243 2023-07-04T20:00:58.879 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/243 2023-07-04T20:00:58.879 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:00:58.892 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:58.986 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:00:58.986 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/244) 2023-07-04T20:00:59.188 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:00:59.192 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:00:59.193 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:00:59.193 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:00:59.193 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:59.526 INFO:teuthology.orchestra.run.smithi077.stdout:generic/244 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:00:59.530 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/244 2023-07-04T20:00:59.530 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/244 2023-07-04T20:00:59.531 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:00:59.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:00:59.560 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:00:59.561 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/245) 2023-07-04T20:00:59.773 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:00:59.774 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:00:59.775 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:00:59.775 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:00:59.775 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:00.163 INFO:teuthology.orchestra.run.smithi077.stdout:generic/245 1s 2023-07-04T20:01:00.167 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/245 2023-07-04T20:01:00.168 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:00.172 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:00.474 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:00.474 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/246) 2023-07-04T20:01:00.751 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:00.755 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:00.756 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:00.757 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:00.757 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:01.130 INFO:teuthology.orchestra.run.smithi077.stdout:generic/246 1s 2023-07-04T20:01:01.135 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/246 2023-07-04T20:01:01.135 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:01.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:01.478 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:01.479 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/247) 2023-07-04T20:01:01.711 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:01.715 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:01.716 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:01.716 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:01.717 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:07.240 INFO:teuthology.orchestra.run.smithi077.stdout:generic/247 6s 2023-07-04T20:01:07.245 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/247 2023-07-04T20:01:07.245 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:07.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:07.255 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:07.256 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/248) 2023-07-04T20:01:07.468 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:07.473 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:07.473 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:07.474 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:07.474 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:07.842 INFO:teuthology.orchestra.run.smithi077.stdout:generic/248 0s 2023-07-04T20:01:07.860 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/248 2023-07-04T20:01:07.861 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:07.862 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:07.896 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:07.897 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/249) 2023-07-04T20:01:08.103 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:08.107 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:08.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:08.662 INFO:teuthology.orchestra.run.smithi077.stdout:generic/249 0s 2023-07-04T20:01:08.666 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/249 2023-07-04T20:01:08.666 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:08.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:08.807 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:08.807 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/250) 2023-07-04T20:01:09.241 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:09.245 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:09.246 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:09.246 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:09.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:09.621 INFO:teuthology.orchestra.run.smithi077.stdout:generic/250 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:01:09.625 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/250 2023-07-04T20:01:09.625 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/250 2023-07-04T20:01:09.625 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:09.627 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:09.650 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:09.651 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/251) 2023-07-04T20:01:09.853 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:09.858 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:09.858 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:09.859 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:09.859 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:10.362 INFO:teuthology.orchestra.run.smithi077.stdout:generic/251 [not run] FITRIM not supported on /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:10.379 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/251 2023-07-04T20:01:10.380 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/251 2023-07-04T20:01:10.380 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:10.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:10.405 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:10.406 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/252) 2023-07-04T20:01:10.619 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:10.624 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:10.626 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:10.626 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:10.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:10.980 INFO:teuthology.orchestra.run.smithi077.stdout:generic/252 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:01:10.985 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/252 2023-07-04T20:01:10.985 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/252 2023-07-04T20:01:10.985 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:10.987 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:11.010 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:11.010 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/253) 2023-07-04T20:01:11.212 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:11.216 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:11.217 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:11.218 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:11.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:12.077 INFO:teuthology.orchestra.run.smithi077.stdout:generic/253 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:01:12.106 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/253 2023-07-04T20:01:12.106 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/253 2023-07-04T20:01:12.106 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:12.109 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:12.147 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:12.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/254) 2023-07-04T20:01:12.353 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:12.356 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:12.357 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:12.358 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:12.358 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:13.150 INFO:teuthology.orchestra.run.smithi077.stdout:generic/254 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:01:13.158 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/254 2023-07-04T20:01:13.158 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/254 2023-07-04T20:01:13.159 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:13.161 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:13.282 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:13.282 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/255) 2023-07-04T20:01:13.515 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:13.519 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:13.520 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:13.520 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:13.521 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:13.971 INFO:teuthology.orchestra.run.smithi077.stdout:generic/255 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:01:13.974 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/255 2023-07-04T20:01:13.975 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/255 2023-07-04T20:01:13.975 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:13.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:14.346 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:14.346 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/256) 2023-07-04T20:01:14.561 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:14.565 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:14.566 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:14.567 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:14.567 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:15.094 INFO:teuthology.orchestra.run.smithi077.stdout:generic/256 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:01:15.098 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/256 2023-07-04T20:01:15.098 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/256 2023-07-04T20:01:15.099 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:15.101 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:15.132 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:15.132 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/257) 2023-07-04T20:01:15.368 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:15.372 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:15.373 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:15.374 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:15.374 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:16.287 INFO:teuthology.orchestra.run.smithi077.stdout:generic/257 1s 2023-07-04T20:01:16.292 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/257 2023-07-04T20:01:16.293 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:16.294 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:16.705 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:16.706 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/258) 2023-07-04T20:01:17.044 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:17.047 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:17.048 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:17.049 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:17.049 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:17.356 INFO:teuthology.orchestra.run.smithi077.stdout:generic/258 [not run] ceph does not support negative timestamps 2023-07-04T20:01:17.360 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/258 2023-07-04T20:01:17.360 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/258 2023-07-04T20:01:17.361 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:17.362 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:17.389 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:17.390 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/259) 2023-07-04T20:01:17.623 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:17.627 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:17.628 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:17.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:generic/259 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:01:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/259 2023-07-04T20:01:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/259 2023-07-04T20:01:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:18.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:18.562 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:18.563 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/260) 2023-07-04T20:01:18.759 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:18.764 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:18.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:19.265 INFO:teuthology.orchestra.run.smithi077.stdout:generic/260 [not run] FITRIM not supported on /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:19.283 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/260 2023-07-04T20:01:19.283 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/260 2023-07-04T20:01:19.283 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:19.286 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:19.312 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:19.313 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/261) 2023-07-04T20:01:19.528 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:19.532 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:19.532 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:19.533 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:19.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:20.241 INFO:teuthology.orchestra.run.smithi077.stdout:generic/261 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:01:20.255 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/261 2023-07-04T20:01:20.255 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/261 2023-07-04T20:01:20.255 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:20.258 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:20.367 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:20.368 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/262) 2023-07-04T20:01:20.609 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:20.613 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:20.614 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:20.615 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:20.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:21.279 INFO:teuthology.orchestra.run.smithi077.stdout:generic/262 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:01:21.320 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/262 2023-07-04T20:01:21.320 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/262 2023-07-04T20:01:21.321 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:01:21.323 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:21.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:01:21.512 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/263) 2023-07-04T20:01:21.711 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:01:21.715 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:01:21.715 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:01:21.716 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:01:21.716 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:01:41.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:01:41.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:01:41.637 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T20:01:41.635+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 144022) UID: 0 2023-07-04T20:01:41.638 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T20:01:41.635+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 144022) UID: 0 2023-07-04T20:02:02.363 INFO:teuthology.orchestra.run.smithi077.stdout:generic/263 41s 2023-07-04T20:02:02.368 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/263 2023-07-04T20:02:02.369 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:02.370 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:02.674 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:02.674 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/264) 2023-07-04T20:02:02.950 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:02.954 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:02.954 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:02.956 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:02.956 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:03.696 INFO:teuthology.orchestra.run.smithi077.stdout:generic/264 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:03.702 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/264 2023-07-04T20:02:03.702 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/264 2023-07-04T20:02:03.702 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:03.705 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:03.779 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:03.779 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/265) 2023-07-04T20:02:04.021 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:04.026 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:04.026 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:04.027 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:04.027 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:04.761 INFO:teuthology.orchestra.run.smithi077.stdout:generic/265 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:04.796 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/265 2023-07-04T20:02:04.797 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/265 2023-07-04T20:02:04.797 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:04.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:04.938 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:04.939 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/266) 2023-07-04T20:02:05.153 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:05.157 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:05.157 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:05.159 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:05.159 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:05.912 INFO:teuthology.orchestra.run.smithi077.stdout:generic/266 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:06.069 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/266 2023-07-04T20:02:06.069 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/266 2023-07-04T20:02:06.069 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:06.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:06.126 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:06.127 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/267) 2023-07-04T20:02:06.355 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:06.358 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:06.359 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:06.360 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:06.360 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:07.214 INFO:teuthology.orchestra.run.smithi077.stdout:generic/267 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:07.243 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/267 2023-07-04T20:02:07.243 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/267 2023-07-04T20:02:07.244 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:07.250 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:07.308 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:07.309 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/268) 2023-07-04T20:02:07.522 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:07.526 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:07.526 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:07.527 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:07.527 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:generic/268 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:08.421 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/268 2023-07-04T20:02:08.421 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/268 2023-07-04T20:02:08.422 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:08.423 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:08.475 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:08.475 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/269) 2023-07-04T20:02:08.688 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:08.691 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:08.691 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:08.692 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:08.692 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:09.031 INFO:teuthology.orchestra.run.smithi077.stdout:generic/269 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:02:09.035 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/269 2023-07-04T20:02:09.036 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/269 2023-07-04T20:02:09.036 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:09.038 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:09.066 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:09.067 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/270) 2023-07-04T20:02:09.288 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:09.292 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:09.292 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:09.293 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:09.293 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:09.610 INFO:teuthology.orchestra.run.smithi077.stdout:generic/270 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:02:09.614 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/270 2023-07-04T20:02:09.614 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/270 2023-07-04T20:02:09.614 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:09.616 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:09.649 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:09.649 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/271) 2023-07-04T20:02:09.853 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:09.857 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:09.858 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:09.858 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:09.859 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:10.673 INFO:teuthology.orchestra.run.smithi077.stdout:generic/271 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:10.715 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/271 2023-07-04T20:02:10.716 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/271 2023-07-04T20:02:10.716 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:10.727 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:10.790 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:10.790 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/272) 2023-07-04T20:02:11.024 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:11.028 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:11.029 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:11.030 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:11.030 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:11.903 INFO:teuthology.orchestra.run.smithi077.stdout:generic/272 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:11.927 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/272 2023-07-04T20:02:11.928 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/272 2023-07-04T20:02:11.928 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:12.001 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:12.002 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/273) 2023-07-04T20:02:12.213 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:12.217 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:12.218 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:12.219 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:12.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:12.578 INFO:teuthology.orchestra.run.smithi077.stdout:generic/273 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:02:12.582 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/273 2023-07-04T20:02:12.582 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/273 2023-07-04T20:02:12.582 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:12.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:12.608 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:12.609 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/274) 2023-07-04T20:02:12.844 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:12.848 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:12.849 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:12.850 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:12.850 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:13.208 INFO:teuthology.orchestra.run.smithi077.stdout:generic/274 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T20:02:13.213 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/274 2023-07-04T20:02:13.213 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/274 2023-07-04T20:02:13.214 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:13.214 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:13.243 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:13.244 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/275) 2023-07-04T20:02:13.477 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:13.481 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:13.482 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:13.483 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:13.483 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:13.811 INFO:teuthology.orchestra.run.smithi077.stdout:generic/275 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:02:13.815 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/275 2023-07-04T20:02:13.815 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/275 2023-07-04T20:02:13.815 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:13.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:13.842 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:13.843 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/276) 2023-07-04T20:02:14.056 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:14.060 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:14.061 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:14.062 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:14.062 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:14.937 INFO:teuthology.orchestra.run.smithi077.stdout:generic/276 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:14.963 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/276 2023-07-04T20:02:14.963 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/276 2023-07-04T20:02:14.963 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:14.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:15.002 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:15.002 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/277) 2023-07-04T20:02:15.217 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:15.221 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:15.221 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:15.222 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:15.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:15.580 INFO:teuthology.orchestra.run.smithi077.stdout:generic/277 [not run] file system doesn't support chattr +A 2023-07-04T20:02:15.584 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/277 2023-07-04T20:02:15.585 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/277 2023-07-04T20:02:15.585 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:15.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:15.625 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:15.626 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/278) 2023-07-04T20:02:15.829 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:15.833 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:15.835 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:16.656 INFO:teuthology.orchestra.run.smithi077.stdout:generic/278 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:16.663 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/278 2023-07-04T20:02:16.664 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/278 2023-07-04T20:02:16.664 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:16.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:16.761 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:16.761 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/279) 2023-07-04T20:02:16.969 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:16.973 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:16.974 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:16.974 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:16.975 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:17.852 INFO:teuthology.orchestra.run.smithi077.stdout:generic/279 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:17.861 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/279 2023-07-04T20:02:17.862 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/279 2023-07-04T20:02:17.862 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:17.864 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:17.915 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:17.915 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/280) 2023-07-04T20:02:18.134 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:18.138 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:18.138 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:18.139 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:18.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:18.489 INFO:teuthology.orchestra.run.smithi077.stdout:generic/280 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:02:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/280 2023-07-04T20:02:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/280 2023-07-04T20:02:18.493 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:18.495 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:18.519 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:18.520 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/281) 2023-07-04T20:02:18.751 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:18.755 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:18.756 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:18.756 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:18.757 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:19.549 INFO:teuthology.orchestra.run.smithi077.stdout:generic/281 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:19.555 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/281 2023-07-04T20:02:19.555 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/281 2023-07-04T20:02:19.555 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:19.571 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:19.679 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:19.680 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/282) 2023-07-04T20:02:19.919 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:19.923 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:19.923 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:19.924 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:19.924 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:20.682 INFO:teuthology.orchestra.run.smithi077.stdout:generic/282 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:20.819 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/282 2023-07-04T20:02:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/282 2023-07-04T20:02:20.825 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:20.844 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:20.882 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:20.883 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/283) 2023-07-04T20:02:21.095 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:21.100 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:21.101 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:21.101 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:21.101 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:22.014 INFO:teuthology.orchestra.run.smithi077.stdout:generic/283 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:22.024 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/283 2023-07-04T20:02:22.025 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/283 2023-07-04T20:02:22.025 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:22.034 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:22.078 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:22.079 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/284) 2023-07-04T20:02:22.294 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:22.298 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:22.299 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:22.300 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:22.300 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:23.035 INFO:teuthology.orchestra.run.smithi077.stdout:generic/284 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:23.175 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/284 2023-07-04T20:02:23.175 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/284 2023-07-04T20:02:23.175 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:23.228 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:23.228 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/285) 2023-07-04T20:02:23.464 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:23.468 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:23.468 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:23.470 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:23.470 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:24.800 INFO:teuthology.orchestra.run.smithi077.stdout:generic/285 1s 2023-07-04T20:02:24.805 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/285 2023-07-04T20:02:24.805 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:24.807 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:25.341 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:25.342 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/286) 2023-07-04T20:02:25.729 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:25.742 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:25.743 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:25.744 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:25.744 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:26.235 INFO:teuthology.orchestra.run.smithi077.stdout:generic/286 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:02:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/286 2023-07-04T20:02:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/286 2023-07-04T20:02:26.245 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:26.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:26.276 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:26.277 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/287) 2023-07-04T20:02:26.579 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:26.584 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:26.585 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:26.586 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:26.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:27.658 INFO:teuthology.orchestra.run.smithi077.stdout:generic/287 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:27.703 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/287 2023-07-04T20:02:27.703 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/287 2023-07-04T20:02:27.704 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:27.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:27.769 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:27.770 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/288) 2023-07-04T20:02:28.110 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:28.115 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:28.115 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:28.116 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:28.117 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:28.779 INFO:teuthology.orchestra.run.smithi077.stdout:generic/288 [not run] FITRIM not supported on /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:28.809 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/288 2023-07-04T20:02:28.809 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/288 2023-07-04T20:02:28.809 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:28.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:28.846 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:28.847 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/289) 2023-07-04T20:02:29.141 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:29.146 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:29.146 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:29.146 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:29.147 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:30.148 INFO:teuthology.orchestra.run.smithi077.stdout:generic/289 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:30.159 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/289 2023-07-04T20:02:30.159 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/289 2023-07-04T20:02:30.160 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:30.300 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:30.301 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/290) 2023-07-04T20:02:30.580 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:30.592 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:30.592 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:30.593 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:30.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:31.695 INFO:teuthology.orchestra.run.smithi077.stdout:generic/290 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:31.739 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/290 2023-07-04T20:02:31.739 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/290 2023-07-04T20:02:31.739 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:31.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:31.787 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:31.788 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/291) 2023-07-04T20:02:32.044 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:32.048 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:32.049 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:32.050 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:32.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:32.881 INFO:teuthology.orchestra.run.smithi077.stdout:generic/291 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:32.920 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/291 2023-07-04T20:02:32.920 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/291 2023-07-04T20:02:32.920 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:32.940 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:32.999 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:32.999 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/292) 2023-07-04T20:02:33.223 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:33.228 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:33.228 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:33.229 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:33.229 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:33.965 INFO:teuthology.orchestra.run.smithi077.stdout:generic/292 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:33.982 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/292 2023-07-04T20:02:33.983 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/292 2023-07-04T20:02:33.983 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:34.115 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:34.186 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:34.186 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/293) 2023-07-04T20:02:34.394 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:34.401 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:34.401 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:34.401 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:34.401 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:35.218 INFO:teuthology.orchestra.run.smithi077.stdout:generic/293 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:35.305 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/293 2023-07-04T20:02:35.305 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/293 2023-07-04T20:02:35.305 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:35.317 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:35.353 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:35.354 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/294) 2023-07-04T20:02:35.587 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:35.591 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:35.592 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:35.592 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:35.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:36.494 INFO:teuthology.orchestra.run.smithi077.stdout:generic/294 1s 2023-07-04T20:02:36.555 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/294 2023-07-04T20:02:36.556 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:36.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:36.571 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:36.572 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/295) 2023-07-04T20:02:36.814 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:36.818 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:36.819 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:36.819 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:36.820 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:37.696 INFO:teuthology.orchestra.run.smithi077.stdout:generic/295 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:37.714 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/295 2023-07-04T20:02:37.714 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/295 2023-07-04T20:02:37.715 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:37.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:37.765 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:37.765 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/296) 2023-07-04T20:02:37.995 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:37.999 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:38.000 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:38.001 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:38.002 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:38.623 INFO:teuthology.orchestra.run.smithi077.stdout:generic/296 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:38.629 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/296 2023-07-04T20:02:38.629 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/296 2023-07-04T20:02:38.630 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:38.632 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:38.918 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:38.919 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/297) 2023-07-04T20:02:39.134 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:39.138 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:39.139 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:39.139 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:39.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:39.966 INFO:teuthology.orchestra.run.smithi077.stdout:generic/297 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:40.055 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/297 2023-07-04T20:02:40.056 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/297 2023-07-04T20:02:40.056 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:40.064 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:40.095 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:40.096 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/298) 2023-07-04T20:02:40.305 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:40.309 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:40.311 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:40.312 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:40.312 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:41.077 INFO:teuthology.orchestra.run.smithi077.stdout:generic/298 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:41.096 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/298 2023-07-04T20:02:41.096 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/298 2023-07-04T20:02:41.096 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:41.134 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:41.230 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:41.230 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/299) 2023-07-04T20:02:41.480 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:41.484 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:41.484 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:41.485 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:41.485 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:41.895 INFO:teuthology.orchestra.run.smithi077.stdout:generic/299 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:02:41.899 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/299 2023-07-04T20:02:41.900 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/299 2023-07-04T20:02:41.900 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:41.902 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:41.933 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:41.934 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/300) 2023-07-04T20:02:42.138 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:42.143 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:42.144 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:42.144 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:42.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:42.547 INFO:teuthology.orchestra.run.smithi077.stdout:generic/300 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:02:42.551 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/300 2023-07-04T20:02:42.551 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/300 2023-07-04T20:02:42.551 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:42.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:42.593 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:42.593 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/301) 2023-07-04T20:02:42.803 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:42.808 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:42.808 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:42.809 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:42.809 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:43.568 INFO:teuthology.orchestra.run.smithi077.stdout:generic/301 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:43.590 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/301 2023-07-04T20:02:43.590 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/301 2023-07-04T20:02:43.591 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:43.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:43.711 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:43.712 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/302) 2023-07-04T20:02:43.925 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:43.928 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:43.929 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:43.930 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:43.930 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:44.682 INFO:teuthology.orchestra.run.smithi077.stdout:generic/302 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:44.708 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/302 2023-07-04T20:02:44.709 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/302 2023-07-04T20:02:44.709 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:44.712 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:44.851 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:44.852 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/303) 2023-07-04T20:02:45.071 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:45.076 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:45.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:45.471 INFO:teuthology.orchestra.run.smithi077.stdout:generic/303 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T20:02:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/303 2023-07-04T20:02:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/303 2023-07-04T20:02:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:45.476 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:45.716 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:45.717 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/304) 2023-07-04T20:02:46.048 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:46.052 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:46.053 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:46.053 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:46.054 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:46.425 INFO:teuthology.orchestra.run.smithi077.stdout:generic/304 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T20:02:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/304 2023-07-04T20:02:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/304 2023-07-04T20:02:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:46.432 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:46.698 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:46.698 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/305) 2023-07-04T20:02:47.014 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:47.019 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:47.021 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:47.021 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:47.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:47.895 INFO:teuthology.orchestra.run.smithi077.stdout:generic/305 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:02:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/305 2023-07-04T20:02:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/305 2023-07-04T20:02:47.951 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:47.952 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:47.984 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:47.985 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/306) 2023-07-04T20:02:48.200 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:48.204 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:48.205 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:48.205 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:48.206 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:49.192 INFO:teuthology.orchestra.run.smithi077.stdout:generic/306 1s 2023-07-04T20:02:49.198 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/306 2023-07-04T20:02:49.198 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:49.200 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:49.430 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:49.431 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/307) 2023-07-04T20:02:49.849 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:49.853 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:49.854 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:49.854 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:49.855 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:51.719 INFO:teuthology.orchestra.run.smithi077.stdout:generic/307 1s 2023-07-04T20:02:51.724 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/307 2023-07-04T20:02:51.724 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:51.726 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:51.734 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:51.734 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/308) 2023-07-04T20:02:51.941 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:51.944 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:51.944 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:51.945 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:51.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:52.304 INFO:teuthology.orchestra.run.smithi077.stdout:generic/308 0s 2023-07-04T20:02:52.309 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/308 2023-07-04T20:02:52.310 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:52.311 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:52.346 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:52.347 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/309) 2023-07-04T20:02:52.557 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:52.561 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:52.562 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:52.562 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:52.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:53.956 INFO:teuthology.orchestra.run.smithi077.stdout:generic/309 1s 2023-07-04T20:02:53.961 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/309 2023-07-04T20:02:53.962 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:02:53.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:02:54.221 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:02:54.222 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/310) 2023-07-04T20:02:54.554 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:02:54.559 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:02:54.559 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:02:54.559 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:02:54.560 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:11.958 INFO:teuthology.orchestra.run.smithi077.stdout:generic/310 77s 2023-07-04T20:04:11.976 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/310 2023-07-04T20:04:11.976 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:11.985 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:12.472 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:12.472 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/311) 2023-07-04T20:04:12.733 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:12.737 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:12.737 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:12.738 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:13.129 INFO:teuthology.orchestra.run.smithi077.stdout:generic/311 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:04:13.133 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/311 2023-07-04T20:04:13.133 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/311 2023-07-04T20:04:13.133 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:13.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:13.167 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:13.168 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/312) 2023-07-04T20:04:13.371 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:13.375 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:13.375 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:13.376 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:13.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:13.727 INFO:teuthology.orchestra.run.smithi077.stdout:generic/312 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:04:13.731 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/312 2023-07-04T20:04:13.731 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/312 2023-07-04T20:04:13.732 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:13.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:13.761 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:13.762 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/313) 2023-07-04T20:04:13.977 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:13.982 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:13.982 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:13.983 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:13.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:18.355 INFO:teuthology.orchestra.run.smithi077.stdout:generic/313 4s 2023-07-04T20:04:18.361 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/313 2023-07-04T20:04:18.362 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:18.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:18.393 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:18.394 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/314) 2023-07-04T20:04:18.603 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:18.608 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:18.608 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:18.609 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:18.609 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:19.070 INFO:teuthology.orchestra.run.smithi077.stdout:generic/314 1s 2023-07-04T20:04:19.075 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/314 2023-07-04T20:04:19.075 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:19.077 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:19.114 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:19.114 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/315) 2023-07-04T20:04:19.329 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:19.332 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:19.333 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:19.334 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:19.334 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:19.683 INFO:teuthology.orchestra.run.smithi077.stdout:generic/315 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T20:04:19.687 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/315 2023-07-04T20:04:19.687 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/315 2023-07-04T20:04:19.687 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:19.689 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:19.715 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:19.715 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/316) 2023-07-04T20:04:19.926 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:19.930 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:19.931 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:19.931 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:19.932 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:20.375 INFO:teuthology.orchestra.run.smithi077.stdout:generic/316 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T20:04:20.380 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/316 2023-07-04T20:04:20.380 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/316 2023-07-04T20:04:20.380 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:20.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:20.684 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:20.685 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/317) 2023-07-04T20:04:20.979 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:20.983 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:20.984 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:20.985 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:20.985 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:21.969 INFO:teuthology.orchestra.run.smithi077.stdout:generic/317 0s 2023-07-04T20:04:21.974 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/317 2023-07-04T20:04:21.974 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:21.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:21.985 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:21.985 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/318) 2023-07-04T20:04:22.190 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:22.194 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:22.195 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:22.195 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:22.195 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:22.995 INFO:teuthology.orchestra.run.smithi077.stdout:generic/318 0s 2023-07-04T20:04:22.999 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/318 2023-07-04T20:04:23.000 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:23.001 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:23.010 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:23.010 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/319) 2023-07-04T20:04:23.221 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:23.225 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:23.226 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:23.226 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:23.227 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:23.793 INFO:teuthology.orchestra.run.smithi077.stdout:generic/319 0s 2023-07-04T20:04:23.798 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/319 2023-07-04T20:04:23.798 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:23.800 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:23.808 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:23.809 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/320) 2023-07-04T20:04:24.014 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:24.018 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:24.019 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:24.019 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:24.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:24.361 INFO:teuthology.orchestra.run.smithi077.stdout:generic/320 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:04:24.365 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/320 2023-07-04T20:04:24.365 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/320 2023-07-04T20:04:24.365 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:24.367 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:24.391 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:24.391 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/321) 2023-07-04T20:04:24.609 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:24.613 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:24.614 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:24.614 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:24.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:24.964 INFO:teuthology.orchestra.run.smithi077.stdout:generic/321 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:04:24.969 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/321 2023-07-04T20:04:24.969 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/321 2023-07-04T20:04:24.970 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:24.971 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:24.995 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:24.996 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/322) 2023-07-04T20:04:25.204 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:25.208 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:25.208 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:25.209 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:25.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:25.553 INFO:teuthology.orchestra.run.smithi077.stdout:generic/322 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:04:25.557 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/322 2023-07-04T20:04:25.557 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/322 2023-07-04T20:04:25.557 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:04:25.559 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:04:25.588 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:04:25.589 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/323) 2023-07-04T20:04:25.804 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:04:25.808 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:04:25.808 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:04:25.809 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:04:25.809 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:05:09.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:05:09.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:05:09.787 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T20:05:09.784+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 361053) UID: 0 2023-07-04T20:05:09.787 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T20:05:09.785+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 361053) UID: 0 2023-07-04T20:05:09.788 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T20:05:09.785+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 361053) UID: 0 2023-07-04T20:05:09.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T20:05:09.786+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 361053) UID: 0 2023-07-04T20:06:27.309 INFO:teuthology.orchestra.run.smithi077.stdout:generic/323 121s 2023-07-04T20:06:27.331 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/323 2023-07-04T20:06:27.331 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:27.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:27.679 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:27.681 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/324) 2023-07-04T20:06:27.965 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:28.025 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:28.025 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:28.025 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:28.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:28.568 INFO:teuthology.orchestra.run.smithi077.stdout:generic/324 [not run] defragmentation not supported for fstype "ceph" 2023-07-04T20:06:28.572 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/324 2023-07-04T20:06:28.572 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/324 2023-07-04T20:06:28.572 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:28.574 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:28.594 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:28.595 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/325) 2023-07-04T20:06:28.929 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:28.933 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:28.934 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:28.934 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:28.934 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:29.586 INFO:teuthology.orchestra.run.smithi077.stdout:generic/325 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:06:29.590 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/325 2023-07-04T20:06:29.590 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/325 2023-07-04T20:06:29.591 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:29.592 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:29.624 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:29.625 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/326) 2023-07-04T20:06:30.018 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:30.021 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:30.022 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:30.023 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:30.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:30.920 INFO:teuthology.orchestra.run.smithi077.stdout:generic/326 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:30.959 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/326 2023-07-04T20:06:30.959 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/326 2023-07-04T20:06:30.960 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:30.967 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:30.995 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:30.996 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/327) 2023-07-04T20:06:31.237 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:31.241 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:31.242 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:31.242 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:31.242 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:32.075 INFO:teuthology.orchestra.run.smithi077.stdout:generic/327 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:32.081 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/327 2023-07-04T20:06:32.081 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/327 2023-07-04T20:06:32.082 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:32.085 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:32.236 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:32.236 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/328) 2023-07-04T20:06:32.448 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:32.452 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:32.453 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:32.453 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:32.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:33.359 INFO:teuthology.orchestra.run.smithi077.stdout:generic/328 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:33.372 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/328 2023-07-04T20:06:33.372 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/328 2023-07-04T20:06:33.372 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:33.379 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:33.413 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:33.414 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/329) 2023-07-04T20:06:33.626 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:33.630 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:33.630 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:33.631 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:33.631 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:34.565 INFO:teuthology.orchestra.run.smithi077.stdout:generic/329 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:34.580 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/329 2023-07-04T20:06:34.580 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/329 2023-07-04T20:06:34.580 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:34.597 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:34.618 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:34.619 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/330) 2023-07-04T20:06:34.820 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:34.824 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:34.824 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:34.825 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:34.825 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:35.714 INFO:teuthology.orchestra.run.smithi077.stdout:generic/330 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:35.744 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/330 2023-07-04T20:06:35.745 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/330 2023-07-04T20:06:35.745 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:35.748 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:35.771 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:35.772 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/331) 2023-07-04T20:06:35.982 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:35.986 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:35.987 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:35.987 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:35.988 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:36.834 INFO:teuthology.orchestra.run.smithi077.stdout:generic/331 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:36.919 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/331 2023-07-04T20:06:36.919 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/331 2023-07-04T20:06:36.919 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:36.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:36.971 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:36.971 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/332) 2023-07-04T20:06:37.173 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:37.177 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:37.177 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:37.178 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:37.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:38.000 INFO:teuthology.orchestra.run.smithi077.stdout:generic/332 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:38.023 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/332 2023-07-04T20:06:38.024 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/332 2023-07-04T20:06:38.024 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:38.125 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:38.149 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:38.150 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/333) 2023-07-04T20:06:38.379 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:38.382 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:38.383 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:38.384 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:38.384 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:39.247 INFO:teuthology.orchestra.run.smithi077.stdout:generic/333 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:39.306 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/333 2023-07-04T20:06:39.307 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/333 2023-07-04T20:06:39.307 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:39.308 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:39.341 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:39.341 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/334) 2023-07-04T20:06:39.560 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:39.564 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:39.565 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:39.565 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:39.566 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:40.530 INFO:teuthology.orchestra.run.smithi077.stdout:generic/334 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:06:40.578 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/334 2023-07-04T20:06:40.578 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/334 2023-07-04T20:06:40.578 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:40.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:40.602 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:40.602 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/335) 2023-07-04T20:06:40.806 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:40.810 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:40.811 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:40.811 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:40.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:41.287 INFO:teuthology.orchestra.run.smithi077.stdout:generic/335 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:06:41.290 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/335 2023-07-04T20:06:41.291 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/335 2023-07-04T20:06:41.291 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:41.292 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:41.315 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:41.315 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/336) 2023-07-04T20:06:41.521 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:41.525 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:41.525 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:41.526 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:41.526 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:41.991 INFO:teuthology.orchestra.run.smithi077.stdout:generic/336 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:06:41.995 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/336 2023-07-04T20:06:41.995 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/336 2023-07-04T20:06:41.995 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:41.997 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:42.022 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:42.023 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/337) 2023-07-04T20:06:42.254 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:42.258 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:42.258 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:42.259 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:42.259 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:42.860 INFO:teuthology.orchestra.run.smithi077.stdout:generic/337 0s 2023-07-04T20:06:42.864 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/337 2023-07-04T20:06:42.865 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:42.866 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:42.874 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:42.875 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/338) 2023-07-04T20:06:43.083 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:43.087 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:43.087 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:43.088 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:43.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:43.484 INFO:teuthology.orchestra.run.smithi077.stdout:generic/338 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:06:43.488 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/338 2023-07-04T20:06:43.488 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/338 2023-07-04T20:06:43.488 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:06:43.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:06:43.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:06:43.512 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/339) 2023-07-04T20:06:43.723 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:06:43.727 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:06:43.727 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:06:43.728 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:06:43.729 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:38.759 INFO:teuthology.orchestra.run.smithi077.stdout:generic/339 55s 2023-07-04T20:07:38.764 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/339 2023-07-04T20:07:38.764 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:38.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:38.774 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:38.775 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/340) 2023-07-04T20:07:39.020 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:39.024 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:39.024 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:39.025 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:39.025 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:42.405 INFO:teuthology.orchestra.run.smithi077.stdout:generic/340 3s 2023-07-04T20:07:42.437 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/340 2023-07-04T20:07:42.437 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:42.437 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:42.444 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:42.445 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/341) 2023-07-04T20:07:42.986 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:42.991 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:42.993 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:42.994 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:42.994 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:43.385 INFO:teuthology.orchestra.run.smithi077.stdout:generic/341 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:07:43.389 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/341 2023-07-04T20:07:43.389 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/341 2023-07-04T20:07:43.389 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:43.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:43.414 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:43.414 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/342) 2023-07-04T20:07:43.611 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:43.615 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:43.616 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:43.617 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:43.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:43.967 INFO:teuthology.orchestra.run.smithi077.stdout:generic/342 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:07:43.971 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/342 2023-07-04T20:07:43.971 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/342 2023-07-04T20:07:43.971 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:43.973 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:43.995 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:43.996 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/343) 2023-07-04T20:07:44.198 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:44.201 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:44.202 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:44.202 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:44.558 INFO:teuthology.orchestra.run.smithi077.stdout:generic/343 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:07:44.562 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/343 2023-07-04T20:07:44.562 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/343 2023-07-04T20:07:44.562 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:44.564 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:44.588 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:44.588 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/344) 2023-07-04T20:07:44.789 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:44.793 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:44.793 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:44.794 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:44.794 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:50.497 INFO:teuthology.orchestra.run.smithi077.stdout:generic/344 6s 2023-07-04T20:07:50.587 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/344 2023-07-04T20:07:50.588 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:50.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:50.610 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:50.611 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/345) 2023-07-04T20:07:51.187 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:51.198 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:51.199 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:51.200 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:51.200 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:56.994 INFO:teuthology.orchestra.run.smithi077.stdout:generic/345 5s 2023-07-04T20:07:57.005 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/345 2023-07-04T20:07:57.006 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:07:57.015 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:07:57.104 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:07:57.105 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/346) 2023-07-04T20:07:57.355 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:07:57.359 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:07:57.360 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:07:57.361 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:07:57.361 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:00.670 INFO:teuthology.orchestra.run.smithi077.stdout:generic/346 3s 2023-07-04T20:08:00.682 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/346 2023-07-04T20:08:00.682 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:00.685 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:00.694 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:00.695 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/347) 2023-07-04T20:08:01.096 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:01.100 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:01.101 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:01.101 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:01.102 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:01.580 INFO:teuthology.orchestra.run.smithi077.stdout:generic/347 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:08:01.584 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/347 2023-07-04T20:08:01.585 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/347 2023-07-04T20:08:01.585 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:01.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:01.621 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:01.622 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/348) 2023-07-04T20:08:01.825 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:01.830 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:01.830 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:01.831 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:01.831 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:02.214 INFO:teuthology.orchestra.run.smithi077.stdout:generic/348 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:08:02.218 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/348 2023-07-04T20:08:02.218 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/348 2023-07-04T20:08:02.218 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:02.220 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:02.244 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:02.245 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/349) 2023-07-04T20:08:02.449 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:02.453 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:02.454 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:02.454 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:02.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:02.911 INFO:teuthology.orchestra.run.smithi077.stdout:generic/349 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-07-04T20:08:02.915 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/349 2023-07-04T20:08:02.916 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/349 2023-07-04T20:08:02.916 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:02.917 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:02.949 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:02.949 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/350) 2023-07-04T20:08:03.160 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:03.164 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:03.164 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:03.165 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:03.166 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:04.082 INFO:teuthology.orchestra.run.smithi077.stdout:generic/350 1s 2023-07-04T20:08:04.086 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/350 2023-07-04T20:08:04.087 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:04.089 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:04.097 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:04.098 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/351) 2023-07-04T20:08:04.338 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:04.342 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:04.343 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:04.344 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:04.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:04.689 INFO:teuthology.orchestra.run.smithi077.stdout:generic/351 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:08:04.693 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/351 2023-07-04T20:08:04.693 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/351 2023-07-04T20:08:04.693 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:04.695 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:04.732 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:04.733 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/352) 2023-07-04T20:08:04.940 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:04.944 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:04.945 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:04.946 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:04.946 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:05.782 INFO:teuthology.orchestra.run.smithi077.stdout:generic/352 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:05.840 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/352 2023-07-04T20:08:05.840 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/352 2023-07-04T20:08:05.841 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:05.844 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:05.915 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:05.915 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/353) 2023-07-04T20:08:06.121 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:06.125 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:06.126 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:06.127 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:06.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:06.941 INFO:teuthology.orchestra.run.smithi077.stdout:generic/353 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:06.976 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/353 2023-07-04T20:08:06.976 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/353 2023-07-04T20:08:06.976 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:06.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:07.041 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:07.042 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/354) 2023-07-04T20:08:07.248 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:07.252 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:07.252 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:07.253 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:07.253 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:12.907 INFO:teuthology.orchestra.run.smithi077.stdout:generic/354 5s 2023-07-04T20:08:12.914 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/354 2023-07-04T20:08:12.914 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:12.920 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:12.929 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:12.929 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/355) 2023-07-04T20:08:13.133 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:13.137 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:13.138 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:13.139 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:13.139 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:14.061 INFO:teuthology.orchestra.run.smithi077.stdout:generic/355 1s 2023-07-04T20:08:14.067 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/355 2023-07-04T20:08:14.067 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:14.069 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:14.402 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:14.403 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/356) 2023-07-04T20:08:14.807 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:14.811 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:14.812 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:14.813 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:14.813 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:15.493 INFO:teuthology.orchestra.run.smithi077.stdout:generic/356 [not run] swapfiles are not supported 2023-07-04T20:08:15.497 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/356 2023-07-04T20:08:15.497 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/356 2023-07-04T20:08:15.497 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:15.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:15.526 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:15.638 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/357) 2023-07-04T20:08:15.843 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:15.847 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:15.847 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:15.848 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:15.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:16.418 INFO:teuthology.orchestra.run.smithi077.stdout:generic/357 [not run] swapfiles are not supported 2023-07-04T20:08:16.422 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/357 2023-07-04T20:08:16.422 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/357 2023-07-04T20:08:16.422 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:16.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:16.447 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:16.448 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/358) 2023-07-04T20:08:16.666 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:16.671 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:16.671 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:16.672 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:16.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:17.430 INFO:teuthology.orchestra.run.smithi077.stdout:generic/358 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:17.511 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/358 2023-07-04T20:08:17.511 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/358 2023-07-04T20:08:17.511 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:17.561 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:17.606 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:17.606 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/359) 2023-07-04T20:08:17.811 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:17.815 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:17.816 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:17.817 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:17.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:18.590 INFO:teuthology.orchestra.run.smithi077.stdout:generic/359 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:18.702 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/359 2023-07-04T20:08:18.703 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/359 2023-07-04T20:08:18.703 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:18.707 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:18.737 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:18.738 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/360) 2023-07-04T20:08:18.946 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:18.951 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:18.951 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:18.952 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:18.952 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:19.320 INFO:teuthology.orchestra.run.smithi077.stdout:generic/360 0s 2023-07-04T20:08:19.330 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/360 2023-07-04T20:08:19.330 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:19.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:19.360 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:19.361 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/361) 2023-07-04T20:08:19.592 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:19.597 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:19.597 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:19.598 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:19.598 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:19.961 INFO:teuthology.orchestra.run.smithi077.stdout:generic/361 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:08:19.964 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/361 2023-07-04T20:08:19.965 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/361 2023-07-04T20:08:19.965 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:19.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:19.992 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:19.993 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/362) 2023-07-04T20:08:20.201 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:20.205 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:20.205 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:20.206 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:20.206 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:20.520 INFO:teuthology.orchestra.run.smithi077.stdout:generic/362 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:20.524 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/362 2023-07-04T20:08:20.524 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/362 2023-07-04T20:08:20.524 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:20.527 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:20.556 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:20.556 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/363) 2023-07-04T20:08:20.774 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:20.779 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:20.779 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:20.780 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:20.780 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:21.112 INFO:teuthology.orchestra.run.smithi077.stdout:generic/363 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:21.116 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/363 2023-07-04T20:08:21.116 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/363 2023-07-04T20:08:21.117 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:21.118 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:21.153 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:21.153 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/364) 2023-07-04T20:08:21.377 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:21.381 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:21.381 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:21.382 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:21.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:21.716 INFO:teuthology.orchestra.run.smithi077.stdout:generic/364 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:21.720 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/364 2023-07-04T20:08:21.720 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/364 2023-07-04T20:08:21.721 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:21.722 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:21.752 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:21.753 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/365) 2023-07-04T20:08:21.973 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:21.977 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:21.978 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:21.979 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:21.979 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:22.316 INFO:teuthology.orchestra.run.smithi077.stdout:generic/365 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:22.319 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/365 2023-07-04T20:08:22.319 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/365 2023-07-04T20:08:22.319 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:22.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:22.345 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:22.345 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/366) 2023-07-04T20:08:22.581 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:22.586 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:22.586 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:22.587 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:22.587 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:22.912 INFO:teuthology.orchestra.run.smithi077.stdout:generic/366 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:22.916 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/366 2023-07-04T20:08:22.916 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/366 2023-07-04T20:08:22.916 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:22.918 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:22.941 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:22.941 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/367) 2023-07-04T20:08:23.151 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:23.155 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:23.156 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:23.157 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:23.483 INFO:teuthology.orchestra.run.smithi077.stdout:generic/367 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:23.487 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/367 2023-07-04T20:08:23.488 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/367 2023-07-04T20:08:23.488 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:23.489 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:23.515 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:23.516 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/368) 2023-07-04T20:08:23.729 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:23.733 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:23.734 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:23.734 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:23.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:24.062 INFO:teuthology.orchestra.run.smithi077.stdout:generic/368 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:24.066 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/368 2023-07-04T20:08:24.067 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/368 2023-07-04T20:08:24.067 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:24.068 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:24.093 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:24.093 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/369) 2023-07-04T20:08:24.302 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:24.305 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:24.306 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:24.307 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:24.307 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:24.627 INFO:teuthology.orchestra.run.smithi077.stdout:generic/369 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:24.631 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/369 2023-07-04T20:08:24.631 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/369 2023-07-04T20:08:24.631 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:24.633 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:24.660 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:24.660 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/370) 2023-07-04T20:08:24.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:08:24.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:08:24.869 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:24.874 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:24.874 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:24.874 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:24.875 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:25.204 INFO:teuthology.orchestra.run.smithi077.stdout:generic/370 [not run] this test requires richacl support on $SCRATCH_DEV 2023-07-04T20:08:25.208 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/370 2023-07-04T20:08:25.209 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/370 2023-07-04T20:08:25.209 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:25.210 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:25.240 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:25.241 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/371) 2023-07-04T20:08:25.446 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:25.450 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:25.450 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:25.451 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:25.451 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:25.792 INFO:teuthology.orchestra.run.smithi077.stdout:generic/371 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:08:25.797 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/371 2023-07-04T20:08:25.797 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/371 2023-07-04T20:08:25.797 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:25.799 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:25.831 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:25.832 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/372) 2023-07-04T20:08:26.053 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:26.057 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:26.058 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:26.059 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:26.059 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:26.885 INFO:teuthology.orchestra.run.smithi077.stdout:generic/372 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:26.892 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/372 2023-07-04T20:08:26.892 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/372 2023-07-04T20:08:26.892 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:26.898 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:26.987 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:26.987 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/373) 2023-07-04T20:08:27.197 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:27.201 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:27.202 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:27.203 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:27.203 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:27.942 INFO:teuthology.orchestra.run.smithi077.stdout:generic/373 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:28.007 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/373 2023-07-04T20:08:28.007 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/373 2023-07-04T20:08:28.007 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:28.114 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:28.164 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:28.165 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/374) 2023-07-04T20:08:28.400 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:28.404 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:28.405 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:28.407 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:28.408 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:28.955 INFO:teuthology.orchestra.run.smithi077.stdout:generic/374 [not run] Dedupe not supported by scratch filesystem type: ceph 2023-07-04T20:08:29.161 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/374 2023-07-04T20:08:29.161 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/374 2023-07-04T20:08:29.162 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:29.185 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:29.337 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:29.337 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/375) 2023-07-04T20:08:29.547 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:29.551 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:29.552 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:29.553 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:29.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:29.983 INFO:teuthology.orchestra.run.smithi077.stdout:generic/375 0s 2023-07-04T20:08:29.987 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/375 2023-07-04T20:08:29.988 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:29.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:30.285 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:30.285 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/376) 2023-07-04T20:08:30.574 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:30.578 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:30.579 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:30.580 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:30.580 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:30.920 INFO:teuthology.orchestra.run.smithi077.stdout:generic/376 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:08:30.924 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/376 2023-07-04T20:08:30.925 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/376 2023-07-04T20:08:30.925 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:30.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:30.957 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:30.957 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/377) 2023-07-04T20:08:31.186 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:31.190 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:31.191 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:31.192 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:31.192 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:31.782 INFO:teuthology.orchestra.run.smithi077.stdout:generic/377 0s 2023-07-04T20:08:31.797 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/377 2023-07-04T20:08:31.797 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:31.799 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:31.808 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:31.809 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/378) 2023-07-04T20:08:32.016 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:32.020 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:32.021 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:32.021 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:32.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:32.500 INFO:teuthology.orchestra.run.smithi077.stdout:generic/378 0s 2023-07-04T20:08:32.505 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/378 2023-07-04T20:08:32.505 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:32.507 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:32.830 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:32.831 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/379) 2023-07-04T20:08:33.090 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:33.094 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:33.095 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:33.095 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:33.096 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:33.454 INFO:teuthology.orchestra.run.smithi077.stdout:generic/379 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:33.459 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/379 2023-07-04T20:08:33.459 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/379 2023-07-04T20:08:33.459 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:33.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:33.481 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:33.482 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/380) 2023-07-04T20:08:33.716 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:33.720 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:33.721 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:33.722 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:33.722 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:34.050 INFO:teuthology.orchestra.run.smithi077.stdout:generic/380 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:34.054 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/380 2023-07-04T20:08:34.054 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/380 2023-07-04T20:08:34.054 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:34.056 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:34.082 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:34.083 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/381) 2023-07-04T20:08:34.284 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:34.289 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:34.289 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:34.290 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:34.290 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:34.626 INFO:teuthology.orchestra.run.smithi077.stdout:generic/381 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:34.630 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/381 2023-07-04T20:08:34.630 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/381 2023-07-04T20:08:34.630 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:34.632 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:34.654 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:34.655 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/382) 2023-07-04T20:08:34.889 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:34.893 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:34.893 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:34.894 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:34.894 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:35.245 INFO:teuthology.orchestra.run.smithi077.stdout:generic/382 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:35.249 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/382 2023-07-04T20:08:35.249 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/382 2023-07-04T20:08:35.249 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:35.251 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:35.280 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:35.280 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/383) 2023-07-04T20:08:35.484 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:35.488 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:35.488 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:35.489 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:35.489 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:35.840 INFO:teuthology.orchestra.run.smithi077.stdout:generic/383 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:35.844 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/383 2023-07-04T20:08:35.844 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/383 2023-07-04T20:08:35.844 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:35.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:35.872 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:35.872 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/384) 2023-07-04T20:08:36.107 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:36.112 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:36.112 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:36.113 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:36.113 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:36.448 INFO:teuthology.orchestra.run.smithi077.stdout:generic/384 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:36.452 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/384 2023-07-04T20:08:36.453 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/384 2023-07-04T20:08:36.453 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:36.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:36.485 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:36.485 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/385) 2023-07-04T20:08:36.688 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:36.693 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:36.693 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:36.694 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:36.694 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:37.042 INFO:teuthology.orchestra.run.smithi077.stdout:generic/385 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:37.046 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/385 2023-07-04T20:08:37.047 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/385 2023-07-04T20:08:37.047 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:37.048 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:37.072 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:37.073 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/386) 2023-07-04T20:08:37.275 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:37.279 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:37.280 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:37.281 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:37.281 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:37.620 INFO:teuthology.orchestra.run.smithi077.stdout:generic/386 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:37.624 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/386 2023-07-04T20:08:37.624 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/386 2023-07-04T20:08:37.625 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:37.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:37.647 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:37.648 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/387) 2023-07-04T20:08:37.851 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:37.855 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:37.856 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:37.856 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:37.857 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:38.693 INFO:teuthology.orchestra.run.smithi077.stdout:generic/387 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:08:38.752 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/387 2023-07-04T20:08:38.752 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/387 2023-07-04T20:08:38.753 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:38.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:38.798 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:38.799 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/388) 2023-07-04T20:08:39.015 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:39.020 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:39.020 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:39.021 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:39.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:39.401 INFO:teuthology.orchestra.run.smithi077.stdout:generic/388 [not run] require 172.21.15.77:6789:/scratch to be local device 2023-07-04T20:08:39.406 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/388 2023-07-04T20:08:39.406 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/388 2023-07-04T20:08:39.406 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:39.408 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:39.439 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:39.439 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/389) 2023-07-04T20:08:39.647 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:39.651 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:39.652 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:39.653 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:39.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:40.000 INFO:teuthology.orchestra.run.smithi077.stdout:generic/389 [not run] O_TMPFILE is not supported 2023-07-04T20:08:40.003 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/389 2023-07-04T20:08:40.004 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/389 2023-07-04T20:08:40.004 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:40.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:40.037 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:40.038 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/390) 2023-07-04T20:08:40.242 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:40.246 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:40.247 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:40.248 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:40.248 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:40.602 INFO:teuthology.orchestra.run.smithi077.stdout:generic/390 [not run] ceph does not support freezing 2023-07-04T20:08:40.605 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/390 2023-07-04T20:08:40.606 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/390 2023-07-04T20:08:40.606 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:40.607 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:40.631 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:40.631 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/391) 2023-07-04T20:08:40.864 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:40.868 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:40.868 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:40.869 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:40.869 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:41.213 INFO:teuthology.orchestra.run.smithi077.stdout:generic/391 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:08:41.217 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/391 2023-07-04T20:08:41.218 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/391 2023-07-04T20:08:41.218 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:41.219 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:41.248 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:41.248 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/392) 2023-07-04T20:08:41.451 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:41.455 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:41.456 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:41.457 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:41.457 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:41.961 INFO:teuthology.orchestra.run.smithi077.stdout:generic/392 [not run] ceph does not support shutdown 2023-07-04T20:08:41.987 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/392 2023-07-04T20:08:41.988 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/392 2023-07-04T20:08:41.988 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:41.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:42.018 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:42.019 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/393) 2023-07-04T20:08:42.222 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:42.226 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:42.227 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:42.228 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:42.228 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:43.282 INFO:teuthology.orchestra.run.smithi077.stdout:generic/393 1s 2023-07-04T20:08:43.286 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/393 2023-07-04T20:08:43.286 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:43.288 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:43.296 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:43.297 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/394) 2023-07-04T20:08:43.530 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:43.534 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:43.535 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:43.536 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:43.536 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:43.926 INFO:teuthology.orchestra.run.smithi077.stdout:generic/394 0s 2023-07-04T20:08:43.931 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/394 2023-07-04T20:08:43.931 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:43.933 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:44.281 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:44.281 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/395) 2023-07-04T20:08:44.510 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:44.514 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:44.515 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:44.516 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:44.516 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:45.337 INFO:teuthology.orchestra.run.smithi077.stdout:generic/395 [not run] kernel does not support ceph encryption 2023-07-04T20:08:45.358 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/395 2023-07-04T20:08:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/395 2023-07-04T20:08:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:45.361 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:45.382 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:45.383 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/396) 2023-07-04T20:08:45.754 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:45.762 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:45.763 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:45.764 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:45.764 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:46.392 INFO:teuthology.orchestra.run.smithi077.stdout:generic/396 [not run] kernel does not support ceph encryption 2023-07-04T20:08:46.416 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/396 2023-07-04T20:08:46.417 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/396 2023-07-04T20:08:46.417 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:46.418 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:46.441 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:46.441 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/397) 2023-07-04T20:08:46.652 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:46.656 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:46.656 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:46.657 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:46.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:47.211 INFO:teuthology.orchestra.run.smithi077.stdout:generic/397 [not run] kernel does not support ceph encryption 2023-07-04T20:08:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/397 2023-07-04T20:08:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/397 2023-07-04T20:08:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:47.232 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:47.255 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:47.256 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/398) 2023-07-04T20:08:47.470 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:47.474 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:47.474 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:47.475 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:47.475 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:generic/398 [not run] kernel does not support ceph encryption 2023-07-04T20:08:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/398 2023-07-04T20:08:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/398 2023-07-04T20:08:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:48.017 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:48.047 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:48.048 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/399) 2023-07-04T20:08:48.262 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:48.266 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:48.266 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:48.267 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:48.267 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:48.859 INFO:teuthology.orchestra.run.smithi077.stdout:generic/399 [not run] kernel does not support ceph encryption 2023-07-04T20:08:48.881 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/399 2023-07-04T20:08:48.882 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/399 2023-07-04T20:08:48.882 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:48.883 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:48.914 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:48.915 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/400) 2023-07-04T20:08:49.119 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:49.123 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:49.124 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:49.125 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:49.125 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:49.454 INFO:teuthology.orchestra.run.smithi077.stdout:generic/400 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T20:08:49.458 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/400 2023-07-04T20:08:49.458 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/400 2023-07-04T20:08:49.459 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:49.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:49.489 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:49.490 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/401) 2023-07-04T20:08:49.699 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:49.703 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:49.704 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:49.704 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:49.705 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:50.795 INFO:teuthology.orchestra.run.smithi077.stdout:generic/401 1s 2023-07-04T20:08:50.807 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/401 2023-07-04T20:08:50.808 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:50.809 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:50.819 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:50.820 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/402) 2023-07-04T20:08:51.050 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:51.055 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:51.055 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:51.056 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:51.057 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:51.612 INFO:teuthology.orchestra.run.smithi077.stdout:generic/402 [not run] filesystem ceph timestamp bounds are unknown 2023-07-04T20:08:51.629 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/402 2023-07-04T20:08:51.629 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/402 2023-07-04T20:08:51.629 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:51.631 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:51.657 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:51.658 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/403) 2023-07-04T20:08:51.865 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:51.870 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:51.871 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:51.871 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:51.872 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:52.615 INFO:teuthology.orchestra.run.smithi077.stdout:generic/403 0s 2023-07-04T20:08:52.620 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/403 2023-07-04T20:08:52.620 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:52.630 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:52.631 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/404) 2023-07-04T20:08:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:53.204 INFO:teuthology.orchestra.run.smithi077.stdout:generic/404 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:08:53.208 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/404 2023-07-04T20:08:53.209 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/404 2023-07-04T20:08:53.209 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:53.210 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:53.240 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:53.241 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/405) 2023-07-04T20:08:53.468 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:53.472 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:53.473 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:53.473 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:53.474 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:53.851 INFO:teuthology.orchestra.run.smithi077.stdout:generic/405 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:08:53.855 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/405 2023-07-04T20:08:53.855 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/405 2023-07-04T20:08:53.855 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:53.857 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:53.886 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:53.886 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/406) 2023-07-04T20:08:54.102 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:54.106 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:54.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:54.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:08:54.920 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:08:55.894 INFO:teuthology.orchestra.run.smithi077.stdout:generic/406 1s 2023-07-04T20:08:55.899 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/406 2023-07-04T20:08:55.899 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:55.901 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:55.910 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:55.911 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/407) 2023-07-04T20:08:56.116 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:56.121 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:56.121 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:56.122 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:56.122 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:56.505 INFO:teuthology.orchestra.run.smithi077.stdout:generic/407 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T20:08:56.508 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/407 2023-07-04T20:08:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/407 2023-07-04T20:08:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:56.511 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:56.721 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:56.722 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/408) 2023-07-04T20:08:57.149 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:57.153 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:57.154 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:57.155 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:57.155 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:57.533 INFO:teuthology.orchestra.run.smithi077.stdout:generic/408 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T20:08:57.536 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/408 2023-07-04T20:08:57.537 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/408 2023-07-04T20:08:57.537 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:57.539 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:57.908 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:57.909 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/409) 2023-07-04T20:08:58.144 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:58.149 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:58.149 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:58.150 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:58.150 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:58.491 INFO:teuthology.orchestra.run.smithi077.stdout:generic/409 [not run] require 172.21.15.77:6789:/scratch to be local device 2023-07-04T20:08:58.495 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/409 2023-07-04T20:08:58.496 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/409 2023-07-04T20:08:58.496 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:58.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:58.520 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:58.520 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/410) 2023-07-04T20:08:58.730 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:58.735 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:58.735 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:58.736 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:58.736 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:59.075 INFO:teuthology.orchestra.run.smithi077.stdout:generic/410 [not run] require 172.21.15.77:6789:/scratch to be local device 2023-07-04T20:08:59.079 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/410 2023-07-04T20:08:59.079 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/410 2023-07-04T20:08:59.080 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:59.081 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:59.102 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:59.103 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/411) 2023-07-04T20:08:59.316 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:59.320 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:59.321 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:59.321 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:59.322 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:59.659 INFO:teuthology.orchestra.run.smithi077.stdout:generic/411 [not run] require 172.21.15.77:6789:/scratch to be local device 2023-07-04T20:08:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/411 2023-07-04T20:08:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/411 2023-07-04T20:08:59.663 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:08:59.665 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:08:59.697 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:08:59.698 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/412) 2023-07-04T20:08:59.914 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:08:59.918 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:08:59.918 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:08:59.919 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:08:59.920 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:00.652 INFO:teuthology.orchestra.run.smithi077.stdout:generic/412 0s 2023-07-04T20:09:00.653 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/412 2023-07-04T20:09:00.653 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:00.653 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:00.660 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:00.661 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/413) 2023-07-04T20:09:00.867 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:00.871 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:00.872 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:00.872 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:00.872 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:01.347 INFO:teuthology.orchestra.run.smithi077.stdout:generic/413 [not run] mount 172.21.15.77:6789:/scratch with dax failed 2023-07-04T20:09:01.351 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/413 2023-07-04T20:09:01.351 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/413 2023-07-04T20:09:01.351 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:01.353 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:01.375 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:01.376 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/414) 2023-07-04T20:09:01.581 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:01.585 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:01.586 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:01.587 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:01.587 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:02.428 INFO:teuthology.orchestra.run.smithi077.stdout:generic/414 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:09:02.461 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/414 2023-07-04T20:09:02.461 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/414 2023-07-04T20:09:02.462 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:02.475 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:02.516 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:02.517 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/415) 2023-07-04T20:09:02.717 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:02.721 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:02.722 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:02.723 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:02.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:03.532 INFO:teuthology.orchestra.run.smithi077.stdout:generic/415 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:09:03.652 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/415 2023-07-04T20:09:03.653 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/415 2023-07-04T20:09:03.653 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:03.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:03.687 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:03.688 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/416) 2023-07-04T20:09:03.896 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:03.900 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:03.901 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:03.902 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:03.902 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:04.267 INFO:teuthology.orchestra.run.smithi077.stdout:generic/416 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:09:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/416 2023-07-04T20:09:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/416 2023-07-04T20:09:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:04.273 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:04.304 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:04.305 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/417) 2023-07-04T20:09:04.513 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:04.517 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:04.518 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:04.519 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:04.519 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:05.026 INFO:teuthology.orchestra.run.smithi077.stdout:generic/417 [not run] ceph does not support shutdown 2023-07-04T20:09:05.042 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/417 2023-07-04T20:09:05.043 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/417 2023-07-04T20:09:05.043 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:05.044 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:05.071 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:05.071 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/418) 2023-07-04T20:09:05.285 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:05.289 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:05.289 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:05.290 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:05.290 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:05.681 INFO:teuthology.orchestra.run.smithi077.stdout:generic/418 [not run] require 172.21.15.77:6789:/test to be valid block disk 2023-07-04T20:09:05.686 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/418 2023-07-04T20:09:05.686 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/418 2023-07-04T20:09:05.686 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:05.688 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:05.721 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:05.721 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/419) 2023-07-04T20:09:05.935 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:05.939 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:05.940 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:05.941 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:05.941 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:06.490 INFO:teuthology.orchestra.run.smithi077.stdout:generic/419 [not run] kernel does not support ceph encryption 2023-07-04T20:09:06.512 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/419 2023-07-04T20:09:06.512 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/419 2023-07-04T20:09:06.512 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:06.513 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:06.537 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:06.538 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/420) 2023-07-04T20:09:06.742 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:06.746 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:06.747 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:06.748 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:06.748 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:07.241 INFO:teuthology.orchestra.run.smithi077.stdout:generic/420 1s 2023-07-04T20:09:07.246 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/420 2023-07-04T20:09:07.246 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:07.248 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:07.577 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:07.578 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/421) 2023-07-04T20:09:07.849 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:07.853 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:07.854 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:07.855 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:07.855 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:08.408 INFO:teuthology.orchestra.run.smithi077.stdout:generic/421 [not run] kernel does not support ceph encryption 2023-07-04T20:09:08.431 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/421 2023-07-04T20:09:08.431 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/421 2023-07-04T20:09:08.432 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:08.433 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:08.455 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:08.455 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/422) 2023-07-04T20:09:08.668 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:08.672 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:08.673 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:08.674 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:08.674 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:09.043 INFO:teuthology.orchestra.run.smithi077.stdout:generic/422 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T20:09:09.047 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/422 2023-07-04T20:09:09.048 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/422 2023-07-04T20:09:09.048 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:09.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:09.082 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:09.083 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/423) 2023-07-04T20:09:09.317 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:09.321 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:09.322 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:09.323 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:09.323 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:09.781 INFO:teuthology.orchestra.run.smithi077.stdout:generic/423 0s 2023-07-04T20:09:09.787 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/423 2023-07-04T20:09:09.787 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:09.789 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:09.888 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:09.889 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/424) 2023-07-04T20:09:10.361 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:10.365 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:10.366 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:10.367 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:10.367 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:10.729 INFO:teuthology.orchestra.run.smithi077.stdout:generic/424 [not run] file system doesn't support any of /bin/chattr +a/+c/+d/+i 2023-07-04T20:09:10.733 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/424 2023-07-04T20:09:10.733 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/424 2023-07-04T20:09:10.734 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:10.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:10.766 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:10.766 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/425) 2023-07-04T20:09:10.972 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:10.977 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:10.977 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:10.978 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:10.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:11.342 INFO:teuthology.orchestra.run.smithi077.stdout:generic/425 [not run] xfs_io fiemap -a failed (old kernel/wrong fs?) 2023-07-04T20:09:11.346 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/425 2023-07-04T20:09:11.347 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/425 2023-07-04T20:09:11.347 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:11.349 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:11.699 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:11.699 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/426) 2023-07-04T20:09:11.937 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:11.941 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:11.942 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:11.942 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:11.943 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:24.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:09:24.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:09:49.962 INFO:teuthology.orchestra.run.smithi077.stdout:generic/426 37s 2023-07-04T20:09:50.054 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/426 2023-07-04T20:09:50.054 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:50.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:50.505 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:50.505 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/427) 2023-07-04T20:09:51.554 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:51.558 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:51.558 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:51.559 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:51.559 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:52.076 INFO:teuthology.orchestra.run.smithi077.stdout:generic/427 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:09:52.080 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/427 2023-07-04T20:09:52.080 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/427 2023-07-04T20:09:52.081 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:52.083 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:52.111 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:52.111 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/428) 2023-07-04T20:09:52.315 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:52.319 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:52.320 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:52.321 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:52.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:generic/428 0s 2023-07-04T20:09:52.692 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/428 2023-07-04T20:09:52.693 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:52.694 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:52.723 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:52.724 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/429) 2023-07-04T20:09:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:52.962 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:53.554 INFO:teuthology.orchestra.run.smithi077.stdout:generic/429 [not run] kernel does not support ceph encryption 2023-07-04T20:09:53.568 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/429 2023-07-04T20:09:53.568 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/429 2023-07-04T20:09:53.569 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:53.570 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:53.597 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:53.598 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/430) 2023-07-04T20:09:53.812 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:53.815 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:53.816 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:53.817 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:53.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:54.280 INFO:teuthology.orchestra.run.smithi077.stdout:generic/430 0s 2023-07-04T20:09:54.285 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/430 2023-07-04T20:09:54.285 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:54.287 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:54.669 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:54.670 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/431) 2023-07-04T20:09:54.904 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:54.908 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:54.909 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:54.910 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:54.910 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:55.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:09:55.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:09:55.338 INFO:teuthology.orchestra.run.smithi077.stdout:generic/431 0s 2023-07-04T20:09:55.342 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/431 2023-07-04T20:09:55.342 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:55.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:55.665 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:55.665 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/432) 2023-07-04T20:09:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:55.944 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:55.945 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:55.946 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:55.946 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:56.368 INFO:teuthology.orchestra.run.smithi077.stdout:generic/432 0s 2023-07-04T20:09:56.372 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/432 2023-07-04T20:09:56.373 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:56.374 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:56.734 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:56.734 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/433) 2023-07-04T20:09:56.971 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:56.975 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:56.975 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:56.976 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:56.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:57.387 INFO:teuthology.orchestra.run.smithi077.stdout:generic/433 0s 2023-07-04T20:09:57.392 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/433 2023-07-04T20:09:57.392 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:57.394 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:57.756 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:57.757 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/434) 2023-07-04T20:09:57.983 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:57.988 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:57.988 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:57.989 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:57.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:58.419 INFO:teuthology.orchestra.run.smithi077.stdout:generic/434 0s 2023-07-04T20:09:58.424 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/434 2023-07-04T20:09:58.424 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:58.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:58.536 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:58.537 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/435) 2023-07-04T20:09:59.029 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:59.033 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:59.034 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:59.034 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:59.035 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:59.579 INFO:teuthology.orchestra.run.smithi077.stdout:generic/435 [not run] kernel does not support ceph encryption 2023-07-04T20:09:59.605 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/435 2023-07-04T20:09:59.605 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/435 2023-07-04T20:09:59.605 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:09:59.607 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:09:59.636 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:09:59.636 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/436) 2023-07-04T20:09:59.867 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:09:59.871 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:09:59.871 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:09:59.873 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:09:59.873 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:10:00.515 INFO:teuthology.orchestra.run.smithi077.stdout:generic/436 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:10:00.519 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/436 2023-07-04T20:10:00.519 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/436 2023-07-04T20:10:00.519 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:10:00.521 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:10:00.648 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:10:00.648 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/437) 2023-07-04T20:10:01.132 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:10:01.135 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:10:01.136 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:10:01.137 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:10:01.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:10:02.130 INFO:teuthology.orchestra.run.smithi077.stdout:generic/437 1s 2023-07-04T20:10:02.135 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/437 2023-07-04T20:10:02.135 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:10:02.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:10:02.163 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:10:02.164 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/438) 2023-07-04T20:10:02.380 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:10:02.384 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:10:02.384 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:10:02.385 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:10:02.385 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:10:25.507 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:10:25.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:10:55.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:10:55.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:11:01.622 INFO:teuthology.orchestra.run.smithi077.stdout:generic/438 59s 2023-07-04T20:11:01.627 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/438 2023-07-04T20:11:01.628 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:01.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:01.659 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:01.659 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/439) 2023-07-04T20:11:01.868 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:01.872 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:01.872 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:01.873 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:01.873 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:03.035 INFO:teuthology.orchestra.run.smithi077.stdout:generic/439 1s 2023-07-04T20:11:03.040 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/439 2023-07-04T20:11:03.040 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:03.042 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:03.070 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:03.071 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/440) 2023-07-04T20:11:03.274 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:03.278 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:03.279 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:03.279 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:03.279 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:03.848 INFO:teuthology.orchestra.run.smithi077.stdout:generic/440 [not run] kernel does not support ceph encryption 2023-07-04T20:11:03.873 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/440 2023-07-04T20:11:03.873 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/440 2023-07-04T20:11:03.873 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:03.875 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:03.899 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:03.899 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/441) 2023-07-04T20:11:04.100 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:04.104 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:04.105 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:04.106 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:04.106 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:04.514 INFO:teuthology.orchestra.run.smithi077.stdout:generic/441 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:11:04.517 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/441 2023-07-04T20:11:04.518 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/441 2023-07-04T20:11:04.518 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:04.520 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:04.542 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:04.543 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/442) 2023-07-04T20:11:04.768 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:04.772 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:04.772 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:04.773 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:04.773 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:05.134 INFO:teuthology.orchestra.run.smithi077.stdout:generic/442 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:11:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/442 2023-07-04T20:11:05.138 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/442 2023-07-04T20:11:05.139 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:05.140 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:05.168 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:05.168 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/443) 2023-07-04T20:11:05.383 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:05.387 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:05.387 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:05.388 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:05.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:05.778 INFO:teuthology.orchestra.run.smithi077.stdout:generic/443 0s 2023-07-04T20:11:05.783 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/443 2023-07-04T20:11:05.783 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:05.785 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:05.818 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:05.818 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/444) 2023-07-04T20:11:06.021 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:06.025 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:06.025 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:06.026 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:06.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:06.473 INFO:teuthology.orchestra.run.smithi077.stdout:generic/444 0s 2023-07-04T20:11:06.477 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/444 2023-07-04T20:11:06.477 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:06.479 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:06.521 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:06.522 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/445) 2023-07-04T20:11:06.727 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:06.731 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:06.732 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:06.733 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:06.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:07.506 INFO:teuthology.orchestra.run.smithi077.stdout:generic/445 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:11:07.510 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/445 2023-07-04T20:11:07.511 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/445 2023-07-04T20:11:07.511 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:07.513 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:07.877 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:07.878 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/446) 2023-07-04T20:11:08.099 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:08.103 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:08.104 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:08.105 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:08.105 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:25.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:11:25.665 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:11:27.733 INFO:teuthology.orchestra.run.smithi077.stdout:generic/446 19s 2023-07-04T20:11:27.756 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/446 2023-07-04T20:11:27.756 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:27.763 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:27.776 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:27.776 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/447) 2023-07-04T20:11:27.983 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:27.987 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:27.988 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:27.989 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:27.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:28.860 INFO:teuthology.orchestra.run.smithi077.stdout:generic/447 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:11:28.918 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/447 2023-07-04T20:11:28.919 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/447 2023-07-04T20:11:28.919 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:28.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:28.972 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:28.973 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/448) 2023-07-04T20:11:29.188 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:29.192 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:29.193 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:29.194 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:29.194 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:30.386 INFO:teuthology.orchestra.run.smithi077.stdout:generic/448 1s 2023-07-04T20:11:30.391 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/448 2023-07-04T20:11:30.391 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:30.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:30.761 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:30.762 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/449) 2023-07-04T20:11:31.024 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:31.028 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:31.029 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:31.029 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:31.414 INFO:teuthology.orchestra.run.smithi077.stdout:generic/449 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T20:11:31.417 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/449 2023-07-04T20:11:31.417 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/449 2023-07-04T20:11:31.417 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:31.419 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:31.770 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:31.771 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/450) 2023-07-04T20:11:32.014 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:32.018 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:32.019 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:32.020 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:32.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:32.741 INFO:teuthology.orchestra.run.smithi077.stdout:generic/450 0s 2023-07-04T20:11:32.746 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/450 2023-07-04T20:11:32.746 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:11:32.748 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:33.052 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:11:33.052 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/451) 2023-07-04T20:11:33.330 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:11:33.334 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:11:33.335 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:11:33.336 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:11:33.336 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:11:56.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:11:56.068 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:12:03.815 INFO:teuthology.orchestra.run.smithi077.stdout:generic/451 30s 2023-07-04T20:12:03.820 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/451 2023-07-04T20:12:03.820 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:03.822 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:04.118 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:04.119 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/452) 2023-07-04T20:12:04.408 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:04.412 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:04.413 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:04.414 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:04.414 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:05.023 INFO:teuthology.orchestra.run.smithi077.stdout:generic/452 1s 2023-07-04T20:12:05.028 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/452 2023-07-04T20:12:05.028 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:05.031 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:05.039 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:05.040 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/453) 2023-07-04T20:12:05.276 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:05.280 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:05.280 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:05.281 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:05.281 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:05.640 INFO:teuthology.orchestra.run.smithi077.stdout:generic/453 [not run] ceph does not allow unrestricted byte streams for names 2023-07-04T20:12:05.644 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/453 2023-07-04T20:12:05.644 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/453 2023-07-04T20:12:05.644 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:05.646 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:05.675 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:05.676 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/454) 2023-07-04T20:12:05.892 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:05.896 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:05.897 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:05.898 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:05.898 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:06.247 INFO:teuthology.orchestra.run.smithi077.stdout:generic/454 [not run] ceph does not allow unrestricted byte streams for names 2023-07-04T20:12:06.251 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/454 2023-07-04T20:12:06.252 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/454 2023-07-04T20:12:06.252 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:06.254 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:06.303 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:06.304 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/455) 2023-07-04T20:12:06.509 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:06.513 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:06.514 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:06.515 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:06.515 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:06.943 INFO:teuthology.orchestra.run.smithi077.stdout:generic/455 [not run] This test requires a valid $LOGWRITES_DEV 2023-07-04T20:12:06.947 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/455 2023-07-04T20:12:06.947 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/455 2023-07-04T20:12:06.947 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:06.949 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:06.980 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:06.981 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/456) 2023-07-04T20:12:07.186 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:07.190 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:07.191 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:07.191 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:07.192 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:07.559 INFO:teuthology.orchestra.run.smithi077.stdout:generic/456 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T20:12:07.563 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/456 2023-07-04T20:12:07.563 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/456 2023-07-04T20:12:07.563 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:07.565 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:07.596 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:07.596 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/457) 2023-07-04T20:12:07.806 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:07.810 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:07.810 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:07.811 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:07.811 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:08.787 INFO:teuthology.orchestra.run.smithi077.stdout:generic/457 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:12:08.792 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/457 2023-07-04T20:12:08.793 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/457 2023-07-04T20:12:08.793 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:08.795 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:08.845 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:08.845 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/458) 2023-07-04T20:12:09.048 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:09.053 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:09.053 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:09.054 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:09.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:09.984 INFO:teuthology.orchestra.run.smithi077.stdout:generic/458 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T20:12:09.996 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/458 2023-07-04T20:12:09.996 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/458 2023-07-04T20:12:09.996 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:09.998 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:10.036 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:10.037 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/459) 2023-07-04T20:12:10.241 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:10.245 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:10.246 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:10.246 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:10.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:10.767 INFO:teuthology.orchestra.run.smithi077.stdout:generic/459 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:12:10.770 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/459 2023-07-04T20:12:10.770 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/459 2023-07-04T20:12:10.770 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:10.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:10.794 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:10.794 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/460) 2023-07-04T20:12:11.007 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:11.011 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:11.012 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:11.013 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:11.013 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:13.048 INFO:teuthology.orchestra.run.smithi077.stdout:generic/460 2s 2023-07-04T20:12:13.052 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/460 2023-07-04T20:12:13.053 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:13.055 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:13.063 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:13.063 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/461) 2023-07-04T20:12:13.266 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:13.270 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:13.271 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:13.272 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:13.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:13.832 INFO:teuthology.orchestra.run.smithi077.stdout:generic/461 [not run] ceph does not support shutdown 2023-07-04T20:12:13.836 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/461 2023-07-04T20:12:13.836 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/461 2023-07-04T20:12:13.836 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:13.837 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:13.873 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:13.874 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/462) 2023-07-04T20:12:14.229 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:14.233 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:14.234 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:14.234 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:14.235 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:14.721 INFO:teuthology.orchestra.run.smithi077.stdout:generic/462 [not run] mount 172.21.15.77:6789:/scratch with dax failed 2023-07-04T20:12:14.726 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/462 2023-07-04T20:12:14.726 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/462 2023-07-04T20:12:14.726 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:14.728 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:14.754 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:14.755 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/463) 2023-07-04T20:12:14.963 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:14.967 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:14.967 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:14.968 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:14.968 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:15.355 INFO:teuthology.orchestra.run.smithi077.stdout:generic/463 [not run] Reflink not supported by test filesystem type: ceph 2023-07-04T20:12:15.360 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/463 2023-07-04T20:12:15.360 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/463 2023-07-04T20:12:15.360 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:12:15.362 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:15.696 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:12:15.696 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/464) 2023-07-04T20:12:15.951 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:12:15.955 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:12:15.955 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:12:15.956 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:12:15.957 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:12:26.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:12:26.152 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:12:56.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:12:56.469 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:13:12.799 INFO:teuthology.orchestra.run.smithi077.stdout:generic/464 56s 2023-07-04T20:13:12.803 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/464 2023-07-04T20:13:12.804 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:13:12.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:12.814 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:13:12.815 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/465) 2023-07-04T20:13:13.017 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:13:13.021 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:13:13.022 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:13:13.023 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:13:13.023 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:17.675 INFO:teuthology.orchestra.run.smithi077.stdout:generic/465 4s 2023-07-04T20:13:17.683 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/465 2023-07-04T20:13:17.685 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:13:17.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:17.693 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:13:17.694 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/466) 2023-07-04T20:13:17.960 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:13:17.964 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:13:17.964 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:13:17.966 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:13:17.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:18.313 INFO:teuthology.orchestra.run.smithi077.stdout:generic/466 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:13:18.317 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/466 2023-07-04T20:13:18.317 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/466 2023-07-04T20:13:18.317 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:13:18.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:18.345 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:13:18.346 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/467) 2023-07-04T20:13:18.561 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:13:18.566 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:13:18.566 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:13:18.567 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:13:18.567 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:13:26.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:13:26.749 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:13:57.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:13:57.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:14:15.771 INFO:teuthology.orchestra.run.smithi077.stdout:generic/467 57s 2023-07-04T20:14:15.837 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/467 2023-07-04T20:14:15.838 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:15.866 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:16.304 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:16.305 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/468) 2023-07-04T20:14:17.312 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:17.316 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:17.316 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:17.317 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:17.317 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:17.879 INFO:teuthology.orchestra.run.smithi077.stdout:generic/468 [not run] ceph does not support shutdown 2023-07-04T20:14:17.899 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/468 2023-07-04T20:14:17.899 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/468 2023-07-04T20:14:17.899 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:17.901 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:17.925 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:17.925 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/469) 2023-07-04T20:14:18.129 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:18.133 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:18.134 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:18.134 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:18.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:18.478 INFO:teuthology.orchestra.run.smithi077.stdout:generic/469 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T20:14:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/469 2023-07-04T20:14:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/469 2023-07-04T20:14:18.483 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:18.485 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:18.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:18.511 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/470) 2023-07-04T20:14:18.730 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:18.734 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:18.735 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:18.736 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:18.736 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:19.147 INFO:teuthology.orchestra.run.smithi077.stdout:generic/470 [not run] This test requires a valid $LOGWRITES_DEV 2023-07-04T20:14:19.151 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/470 2023-07-04T20:14:19.151 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/470 2023-07-04T20:14:19.151 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:19.153 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:19.180 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:19.180 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/471) 2023-07-04T20:14:19.384 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:19.388 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:19.389 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:19.389 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:19.389 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:19.778 INFO:teuthology.orchestra.run.smithi077.stdout:generic/471 [not run] xfs_io pwrite -V 1 -b 4k -N failed (old kernel/wrong fs?) 2023-07-04T20:14:19.781 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/471 2023-07-04T20:14:19.782 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/471 2023-07-04T20:14:19.782 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:19.784 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:20.060 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:20.060 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/472) 2023-07-04T20:14:20.370 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:20.374 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:20.375 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:20.376 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:20.376 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:21.031 INFO:teuthology.orchestra.run.smithi077.stdout:generic/472 [not run] swapfiles are not supported 2023-07-04T20:14:21.035 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/472 2023-07-04T20:14:21.035 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/472 2023-07-04T20:14:21.036 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:21.037 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:21.069 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:21.069 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/473) 2023-07-04T20:14:21.276 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:21.280 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:21.280 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:21.281 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:21.281 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:21.677 INFO:teuthology.orchestra.run.smithi077.stdout:generic/473 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-07-04T20:14:21.680 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/473 2023-07-04T20:14:21.681 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/473 2023-07-04T20:14:21.681 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:21.683 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:21.710 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:21.711 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/474) 2023-07-04T20:14:21.921 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:21.926 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:21.926 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:21.927 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:21.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:22.446 INFO:teuthology.orchestra.run.smithi077.stdout:generic/474 [not run] ceph does not support shutdown 2023-07-04T20:14:22.466 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/474 2023-07-04T20:14:22.467 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/474 2023-07-04T20:14:22.467 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:22.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:22.493 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:22.493 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/475) 2023-07-04T20:14:22.697 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:22.701 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:22.701 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:22.702 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:22.702 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:23.040 INFO:teuthology.orchestra.run.smithi077.stdout:generic/475 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T20:14:23.044 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/475 2023-07-04T20:14:23.044 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/475 2023-07-04T20:14:23.045 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T20:14:23.046 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:23.072 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T20:14:23.073 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/476) 2023-07-04T20:14:23.284 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T20:14:23.287 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T20:14:23.288 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T20:14:23.289 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T20:14:23.289 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T20:14:28.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:14:28.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:14:58.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:14:58.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:15:30.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:15:30.276 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:16:00.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:16:00.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:16:31.900 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:16:31.903 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:17:02.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:17:02.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:17:32.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:17:32.462 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:18:02.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:18:02.566 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:18:32.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:18:32.931 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:19:03.218 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:19:03.229 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:19:33.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:19:33.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:20:04.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:20:04.392 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:20:34.710 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:20:34.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:20:34.793 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T20:20:34.791+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 146636) UID: 0 2023-07-04T20:20:34.797 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T20:20:34.791+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 146636) UID: 0 2023-07-04T20:24:16.578 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T20:24:16.576+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146686) UID: 0 2023-07-04T20:24:16.579 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T20:24:16.576+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146686) UID: 0 2023-07-04T20:24:16.579 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T20:24:16.577+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146686) UID: 0 2023-07-04T20:24:16.582 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T20:24:16.581+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146686) UID: 0 2023-07-04T20:28:30.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:28:30.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:28:30.620 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T20:28:30.617+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638611) UID: 0 2023-07-04T20:28:30.622 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T20:28:30.620+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638611) UID: 0 2023-07-04T20:28:30.623 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T20:28:30.620+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638611) UID: 0 2023-07-04T20:28:30.623 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T20:28:30.620+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638611) UID: 0 2023-07-04T20:32:24.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:32:24.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:32:24.155 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T20:32:24.154+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146828) UID: 0 2023-07-04T20:32:24.156 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T20:32:24.154+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146828) UID: 0 2023-07-04T20:32:24.158 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T20:32:24.157+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146828) UID: 0 2023-07-04T20:32:24.160 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T20:32:24.159+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 146828) UID: 0 2023-07-04T20:36:27.999 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:36:28.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:36:28.096 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T20:36:28.082+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638737) UID: 0 2023-07-04T20:36:28.097 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T20:36:28.082+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638737) UID: 0 2023-07-04T20:36:28.118 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T20:36:28.076+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638737) UID: 0 2023-07-04T20:36:28.119 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T20:36:28.082+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638737) UID: 0 2023-07-04T20:36:28.278 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T20:36:28.276+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 146894) UID: 0 2023-07-04T20:36:28.304 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T20:36:28.276+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 146894) UID: 0 2023-07-04T20:41:12.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:41:12.575 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:41:13.148 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T20:41:13.146+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638809) UID: 0 2023-07-04T20:41:13.151 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T20:41:13.147+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638809) UID: 0 2023-07-04T20:41:13.156 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T20:41:13.147+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638809) UID: 0 2023-07-04T20:41:13.158 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T20:41:13.147+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 638809) UID: 0 2023-07-04T20:44:49.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:44:49.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:44:50.085 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T20:44:50.083+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147023) UID: 0 2023-07-04T20:44:50.098 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T20:44:50.096+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147023) UID: 0 2023-07-04T20:44:50.109 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T20:44:50.096+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147023) UID: 0 2023-07-04T20:44:50.118 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T20:44:50.096+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147023) UID: 0 2023-07-04T20:48:47.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:48:47.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:49:17.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:49:17.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:49:47.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:49:47.912 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:49:48.063 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T20:49:48.062+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147140) UID: 0 2023-07-04T20:49:48.068 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T20:49:48.067+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147140) UID: 0 2023-07-04T20:54:02.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:54:02.013 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:54:02.564 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T20:54:02.561+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 639070) UID: 0 2023-07-04T20:54:02.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T20:54:02.561+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 639070) UID: 0 2023-07-04T20:54:02.586 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T20:54:02.561+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 639070) UID: 0 2023-07-04T20:54:02.586 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T20:54:02.561+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 639070) UID: 0 2023-07-04T20:57:22.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:57:22.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:57:53.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:57:53.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T20:57:53.835 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T20:57:53.834+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147308) UID: 0 2023-07-04T20:57:53.840 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T20:57:53.834+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147308) UID: 0 2023-07-04T20:57:53.842 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T20:57:53.834+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147308) UID: 0 2023-07-04T20:57:53.844 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T20:57:53.834+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147308) UID: 0 2023-07-04T21:01:48.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:01:48.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:01:49.091 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T21:01:49.089+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147383) UID: 0 2023-07-04T21:01:49.101 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T21:01:49.091+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147383) UID: 0 2023-07-04T21:06:10.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:06:10.665 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:06:40.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:06:40.870 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:07:11.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:07:11.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:07:11.268 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T21:07:11.266+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147505) UID: 0 2023-07-04T21:07:11.271 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T21:07:11.270+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147505) UID: 0 2023-07-04T21:07:11.272 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T21:07:11.270+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147505) UID: 0 2023-07-04T21:07:11.272 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:07:11.270+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147505) UID: 0 2023-07-04T21:11:04.329 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:11:04.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:11:35.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:11:35.163 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:12:05.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:12:05.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:12:18.787 INFO:teuthology.orchestra.run.smithi077.stdout:generic/476 3475s 2023-07-04T21:12:18.824 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/476 2023-07-04T21:12:18.824 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:12:18.826 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:12:18.836 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:12:18.836 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/477) 2023-07-04T21:12:31.166 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:12:31.171 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:12:31.171 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:12:31.172 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:12:31.172 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:12:35.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:12:35.324 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:12:35.521 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T21:12:35.520+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147665) UID: 0 2023-07-04T21:12:35.522 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T21:12:35.520+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147665) UID: 0 2023-07-04T21:15:51.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:15:51.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:16:21.379 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:16:21.381 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:16:51.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:16:51.808 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:17:21.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:17:21.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:17:52.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:17:52.130 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:18:22.218 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:18:22.221 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:18:52.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:18:52.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:19:22.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:19:22.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:19:22.454 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T21:19:22.452+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 147888) UID: 0 2023-07-04T21:19:22.455 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T21:19:22.453+0000 7fb0bf5a3700 -1 Fail to open '/proc/147888/cmdline' error = (2) No such file or directory 2023-07-04T21:19:22.455 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T21:19:22.453+0000 7fb0bf5a3700 -1 received signal: Hangup from (PID: 147888) UID: 0 2023-07-04T21:21:56.517 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T21:21:56.515+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 147918) UID: 0 2023-07-04T21:21:56.518 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:21:56.516+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 147918) UID: 0 2023-07-04T21:21:56.519 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T21:21:56.517+0000 7f0e6e9ca700 -1 received signal: Hangup from (PID: 147918) UID: 0 2023-07-04T21:21:56.522 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T21:21:56.520+0000 7f218b0dd700 -1 received signal: Hangup from (PID: 147918) UID: 0 2023-07-04T21:22:31.192 INFO:teuthology.orchestra.run.smithi077.stdout:generic/477 26s 2023-07-04T21:22:31.263 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/477 2023-07-04T21:22:31.263 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:31.291 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:31.438 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:31.439 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/478) 2023-07-04T21:22:32.641 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:32.651 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:32.652 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:32.652 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:32.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:49.821 INFO:teuthology.orchestra.run.smithi077.stdout:generic/478 17s 2023-07-04T21:22:49.827 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/478 2023-07-04T21:22:49.827 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:49.829 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:49.888 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:49.889 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/479) 2023-07-04T21:22:50.234 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:50.241 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:50.241 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:50.241 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:50.242 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:50.876 INFO:teuthology.orchestra.run.smithi077.stdout:generic/479 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:22:50.880 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/479 2023-07-04T21:22:50.881 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/479 2023-07-04T21:22:50.881 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:50.883 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:50.925 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:50.925 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/480) 2023-07-04T21:22:51.358 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:51.362 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:51.362 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:51.363 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:51.363 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:51.894 INFO:teuthology.orchestra.run.smithi077.stdout:generic/480 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:22:51.898 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/480 2023-07-04T21:22:51.899 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/480 2023-07-04T21:22:51.899 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:51.900 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:51.934 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:51.934 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/481) 2023-07-04T21:22:52.278 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:52.281 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:52.282 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:52.283 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:52.283 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:generic/481 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:22:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/481 2023-07-04T21:22:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/481 2023-07-04T21:22:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:52.821 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:52.854 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:52.854 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/482) 2023-07-04T21:22:53.157 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:53.162 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:53.163 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:53.164 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:53.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:53.683 INFO:teuthology.orchestra.run.smithi077.stdout:generic/482 [not run] This test requires a valid $LOGWRITES_DEV 2023-07-04T21:22:53.687 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/482 2023-07-04T21:22:53.687 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/482 2023-07-04T21:22:53.688 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:53.689 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:53.736 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:53.736 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/483) 2023-07-04T21:22:54.088 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:54.094 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:54.095 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:54.096 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:54.096 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:54.574 INFO:teuthology.orchestra.run.smithi077.stdout:generic/483 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:22:54.578 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/483 2023-07-04T21:22:54.579 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/483 2023-07-04T21:22:54.579 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:54.580 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:54.627 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:54.627 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/484) 2023-07-04T21:22:54.935 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:54.939 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:54.940 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:54.941 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:54.941 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:55.446 INFO:teuthology.orchestra.run.smithi077.stdout:generic/484 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:22:55.451 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/484 2023-07-04T21:22:55.452 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/484 2023-07-04T21:22:55.452 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:55.454 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:55.494 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:55.495 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/485) 2023-07-04T21:22:55.804 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:55.809 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:55.810 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:55.810 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:55.811 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:56.300 INFO:teuthology.orchestra.run.smithi077.stdout:generic/485 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T21:22:56.305 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/485 2023-07-04T21:22:56.305 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/485 2023-07-04T21:22:56.305 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:56.307 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:56.362 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:56.363 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/486) 2023-07-04T21:22:56.662 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:56.667 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:56.668 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:56.668 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:56.669 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:58.048 INFO:teuthology.orchestra.run.smithi077.stdout:generic/486 1s 2023-07-04T21:22:58.129 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/486 2023-07-04T21:22:58.130 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:58.132 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:58.147 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:58.148 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/487) 2023-07-04T21:22:58.446 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:58.454 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:58.455 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:58.455 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:58.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:59.010 INFO:teuthology.orchestra.run.smithi077.stdout:generic/487 [not run] This test requires a valid $SCRATCH_LOGDEV 2023-07-04T21:22:59.014 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/487 2023-07-04T21:22:59.015 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/487 2023-07-04T21:22:59.015 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:59.022 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:59.056 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:59.056 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/488) 2023-07-04T21:22:59.394 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:22:59.398 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:22:59.402 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:22:59.403 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:22:59.403 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:59.873 INFO:teuthology.orchestra.run.smithi077.stdout:generic/488 [not run] Filesystem ceph not supported in _scratch_mkfs_sized 2023-07-04T21:22:59.881 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/488 2023-07-04T21:22:59.882 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/488 2023-07-04T21:22:59.882 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:22:59.887 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:22:59.915 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:22:59.916 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/489) 2023-07-04T21:23:00.209 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:00.214 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:00.214 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:00.215 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:00.216 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:00.710 INFO:teuthology.orchestra.run.smithi077.stdout:generic/489 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:00.715 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/489 2023-07-04T21:23:00.715 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/489 2023-07-04T21:23:00.715 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:00.717 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:00.750 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:00.751 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/490) 2023-07-04T21:23:01.116 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:01.127 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:01.128 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:01.129 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:01.130 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:03.509 INFO:teuthology.orchestra.run.smithi077.stdout:generic/490 2s 2023-07-04T21:23:03.515 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/490 2023-07-04T21:23:03.516 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:03.517 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:03.945 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:03.946 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/491) 2023-07-04T21:23:04.298 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:04.309 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:04.310 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:04.310 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:04.311 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:04.789 INFO:teuthology.orchestra.run.smithi077.stdout:generic/491 [not run] ceph does not support freezing 2023-07-04T21:23:04.793 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/491 2023-07-04T21:23:04.793 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/491 2023-07-04T21:23:04.793 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:04.795 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:04.828 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:04.829 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/492) 2023-07-04T21:23:05.151 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:05.155 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:05.156 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:05.157 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:05.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:05.660 INFO:teuthology.orchestra.run.smithi077.stdout:generic/492 [not run] xfs_io label failed (old kernel/wrong fs?) 2023-07-04T21:23:05.666 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/492 2023-07-04T21:23:05.666 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/492 2023-07-04T21:23:05.666 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:05.667 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:05.699 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:05.700 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/493) 2023-07-04T21:23:06.028 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:06.034 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:06.034 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:06.036 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:06.036 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:06.963 INFO:teuthology.orchestra.run.smithi077.stdout:generic/493 [not run] swapfiles are not supported 2023-07-04T21:23:06.967 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/493 2023-07-04T21:23:06.967 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/493 2023-07-04T21:23:06.967 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:06.978 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:07.026 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:07.027 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/494) 2023-07-04T21:23:07.321 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:07.325 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:07.326 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:07.327 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:07.327 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:08.103 INFO:teuthology.orchestra.run.smithi077.stdout:generic/494 [not run] swapfiles are not supported 2023-07-04T21:23:08.108 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/494 2023-07-04T21:23:08.108 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/494 2023-07-04T21:23:08.108 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:08.113 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:08.149 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:08.150 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/495) 2023-07-04T21:23:08.510 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:08.516 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:08.517 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:08.524 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:08.525 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:09.384 INFO:teuthology.orchestra.run.smithi077.stdout:generic/495 [not run] swapfiles are not supported 2023-07-04T21:23:09.389 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/495 2023-07-04T21:23:09.389 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/495 2023-07-04T21:23:09.390 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:09.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:09.440 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:09.440 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/496) 2023-07-04T21:23:09.785 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:09.799 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:09.799 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:09.801 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:09.801 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:10.639 INFO:teuthology.orchestra.run.smithi077.stdout:generic/496 [not run] swapfiles are not supported 2023-07-04T21:23:10.647 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/496 2023-07-04T21:23:10.647 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/496 2023-07-04T21:23:10.648 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:10.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:10.689 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:10.689 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/497) 2023-07-04T21:23:11.011 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:11.024 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:11.025 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:11.026 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:11.026 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:11.857 INFO:teuthology.orchestra.run.smithi077.stdout:generic/497 [not run] swapfiles are not supported 2023-07-04T21:23:11.862 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/497 2023-07-04T21:23:11.863 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/497 2023-07-04T21:23:11.863 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:11.908 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:11.909 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/498) 2023-07-04T21:23:12.233 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:12.237 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:12.239 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:12.239 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:12.239 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:12.764 INFO:teuthology.orchestra.run.smithi077.stdout:generic/498 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:12.769 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/498 2023-07-04T21:23:12.769 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/498 2023-07-04T21:23:12.770 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:12.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:12.828 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:12.828 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/499) 2023-07-04T21:23:13.131 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:13.135 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:13.136 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:13.137 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:13.137 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:13.650 INFO:teuthology.orchestra.run.smithi077.stdout:generic/499 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T21:23:13.660 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/499 2023-07-04T21:23:13.661 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/499 2023-07-04T21:23:13.661 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:13.663 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:13.716 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:13.717 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/500) 2023-07-04T21:23:14.057 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:14.061 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:14.061 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:14.062 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:14.062 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:14.587 INFO:teuthology.orchestra.run.smithi077.stdout:generic/500 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:14.591 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/500 2023-07-04T21:23:14.592 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/500 2023-07-04T21:23:14.592 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:14.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:14.617 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:14.617 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/501) 2023-07-04T21:23:14.926 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:14.932 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:14.933 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:14.935 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:15.937 INFO:teuthology.orchestra.run.smithi077.stdout:generic/501 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T21:23:15.943 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/501 2023-07-04T21:23:15.943 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/501 2023-07-04T21:23:15.943 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:16.092 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:16.141 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:16.141 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/502) 2023-07-04T21:23:16.436 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:16.440 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:16.441 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:16.441 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:16.442 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:17.014 INFO:teuthology.orchestra.run.smithi077.stdout:generic/502 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:17.019 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/502 2023-07-04T21:23:17.020 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/502 2023-07-04T21:23:17.020 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:17.021 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:17.067 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:17.068 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/503) 2023-07-04T21:23:17.394 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:17.398 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:17.399 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:17.400 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:17.400 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:17.861 INFO:teuthology.orchestra.run.smithi077.stdout:generic/503 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T21:23:17.866 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/503 2023-07-04T21:23:17.866 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/503 2023-07-04T21:23:17.867 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:17.868 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:17.910 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:17.911 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/504) 2023-07-04T21:23:18.281 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:18.287 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:18.288 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:18.289 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:18.289 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:18.820 INFO:teuthology.orchestra.run.smithi077.stdout:generic/504 0s 2023-07-04T21:23:18.826 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/504 2023-07-04T21:23:18.826 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:18.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:18.879 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:18.879 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/505) 2023-07-04T21:23:19.175 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:19.182 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:19.183 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:19.183 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:19.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:19.914 INFO:teuthology.orchestra.run.smithi077.stdout:generic/505 [not run] ceph does not support shutdown 2023-07-04T21:23:19.951 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/505 2023-07-04T21:23:19.951 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/505 2023-07-04T21:23:19.951 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:19.953 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:19.984 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:19.985 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/506) 2023-07-04T21:23:20.287 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:20.291 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:20.291 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:20.292 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:20.292 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:20.738 INFO:teuthology.orchestra.run.smithi077.stdout:generic/506 [not run] disk quotas not supported by this filesystem type: ceph 2023-07-04T21:23:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/506 2023-07-04T21:23:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/506 2023-07-04T21:23:20.744 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:20.747 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:20.793 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:20.793 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/507) 2023-07-04T21:23:21.127 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:21.132 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:21.132 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:21.133 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:21.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:21.686 INFO:teuthology.orchestra.run.smithi077.stdout:generic/507 [not run] file system doesn't support chattr +ASai 2023-07-04T21:23:21.698 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/507 2023-07-04T21:23:21.698 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/507 2023-07-04T21:23:21.698 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:21.700 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:21.746 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:21.747 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/508) 2023-07-04T21:23:22.066 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:22.072 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:22.073 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:22.074 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:22.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:22.510 INFO:teuthology.orchestra.run.smithi077.stdout:generic/508 [not run] lsattr not supported by test filesystem type: ceph 2023-07-04T21:23:22.515 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/508 2023-07-04T21:23:22.515 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/508 2023-07-04T21:23:22.515 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:22.523 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:22.571 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:22.572 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/509) 2023-07-04T21:23:22.941 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:22.945 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:22.945 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:22.946 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:22.946 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:23.444 INFO:teuthology.orchestra.run.smithi077.stdout:generic/509 [not run] O_TMPFILE is not supported 2023-07-04T21:23:23.448 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/509 2023-07-04T21:23:23.448 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/509 2023-07-04T21:23:23.448 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:23.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:23.484 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:23.485 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/510) 2023-07-04T21:23:23.822 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:23.825 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:23.826 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:23.827 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:23.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:24.319 INFO:teuthology.orchestra.run.smithi077.stdout:generic/510 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:24.329 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/510 2023-07-04T21:23:24.329 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/510 2023-07-04T21:23:24.329 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:24.331 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:24.379 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:24.379 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/511) 2023-07-04T21:23:24.694 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:24.700 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:24.701 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:24.702 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:24.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:25.222 INFO:teuthology.orchestra.run.smithi077.stdout:generic/511 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) 2023-07-04T21:23:25.227 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/511 2023-07-04T21:23:25.227 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/511 2023-07-04T21:23:25.227 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:25.234 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:25.275 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:25.276 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/512) 2023-07-04T21:23:25.566 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:25.571 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:25.572 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:25.573 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:25.573 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:26.111 INFO:teuthology.orchestra.run.smithi077.stdout:generic/512 [not run] xfs_io falloc failed (old kernel/wrong fs?) 2023-07-04T21:23:26.115 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/512 2023-07-04T21:23:26.115 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/512 2023-07-04T21:23:26.115 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:26.117 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:26.165 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:26.165 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/513) 2023-07-04T21:23:26.487 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:26.491 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:26.491 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:26.492 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:26.492 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:27.610 INFO:teuthology.orchestra.run.smithi077.stdout:generic/513 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T21:23:27.637 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/513 2023-07-04T21:23:27.637 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/513 2023-07-04T21:23:27.637 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:27.640 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:27.695 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:27.695 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/514) 2023-07-04T21:23:28.002 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:28.007 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:28.008 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:28.009 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:28.009 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:29.071 INFO:teuthology.orchestra.run.smithi077.stdout:generic/514 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T21:23:29.078 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/514 2023-07-04T21:23:29.078 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/514 2023-07-04T21:23:29.078 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:29.182 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:29.257 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:29.257 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/515) 2023-07-04T21:23:29.673 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:29.679 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:29.679 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:29.681 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:29.682 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:30.827 INFO:teuthology.orchestra.run.smithi077.stdout:generic/515 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T21:23:30.846 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/515 2023-07-04T21:23:30.846 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/515 2023-07-04T21:23:30.846 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:30.850 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:30.961 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:30.961 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/516) 2023-07-04T21:23:31.250 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:31.254 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:31.254 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:31.255 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:31.255 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:31.862 INFO:teuthology.orchestra.run.smithi077.stdout:generic/516 [not run] Dedupe not supported by test filesystem type: ceph 2023-07-04T21:23:31.866 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/516 2023-07-04T21:23:31.867 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/516 2023-07-04T21:23:31.867 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:31.869 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:32.182 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:32.183 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/517) 2023-07-04T21:23:32.606 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:32.610 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:32.611 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:32.613 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:32.613 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:33.437 INFO:teuthology.orchestra.run.smithi077.stdout:generic/517 [not run] Dedupe not supported by scratch filesystem type: ceph 2023-07-04T21:23:33.812 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/517 2023-07-04T21:23:33.812 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/517 2023-07-04T21:23:33.812 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:33.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:33.902 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:33.903 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/518) 2023-07-04T21:23:34.267 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:34.271 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:34.271 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:34.272 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:34.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:35.343 INFO:teuthology.orchestra.run.smithi077.stdout:generic/518 [not run] Reflink not supported by scratch filesystem type: ceph 2023-07-04T21:23:35.350 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/518 2023-07-04T21:23:35.350 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/518 2023-07-04T21:23:35.350 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:35.431 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:35.542 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:35.543 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/519) 2023-07-04T21:23:35.877 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:35.888 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:35.889 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:35.890 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:35.890 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:36.425 INFO:teuthology.orchestra.run.smithi077.stdout:generic/519 [not run] FIBMAP not supported by this filesystem 2023-07-04T21:23:36.436 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/519 2023-07-04T21:23:36.436 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/519 2023-07-04T21:23:36.437 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:36.439 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:36.731 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:36.731 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/520) 2023-07-04T21:23:37.217 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:37.221 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:37.222 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:37.223 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:37.713 INFO:teuthology.orchestra.run.smithi077.stdout:generic/520 [not run] require 172.21.15.77:6789:/scratch to be valid block disk 2023-07-04T21:23:37.718 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/520 2023-07-04T21:23:37.718 INFO:teuthology.orchestra.run.smithi077.stdout:Not run: generic/520 2023-07-04T21:23:37.718 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T21:23:37.721 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:23:37.773 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T21:23:37.774 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/521) 2023-07-04T21:23:38.113 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T21:23:38.117 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T21:23:38.118 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T21:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T21:23:38.119 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T21:24:58.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:24:58.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:24:58.469 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T21:24:58.464+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 662100) UID: 0 2023-07-04T21:24:58.472 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T21:24:58.470+0000 7f7dfbfa6700 -1 Fail to open '/proc/662100/cmdline' error = (2) No such file or directory 2023-07-04T21:24:58.472 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T21:24:58.470+0000 7f7dfbfa6700 -1 received signal: Hangup from (PID: 662100) UID: 0 2023-07-04T21:24:58.472 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:24:58.471+0000 7f9b4664e700 -1 Fail to open '/proc/662100/cmdline' error = (2) No such file or directory 2023-07-04T21:24:58.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:24:58.471+0000 7f9b4664e700 -1 received signal: Hangup from (PID: 662100) UID: 0 2023-07-04T21:24:58.473 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:24:58.471+0000 7f2bac08a700 -1 Fail to open '/proc/662100/cmdline' error = (2) No such file or directory 2023-07-04T21:24:58.473 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:24:58.471+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 662100) UID: 0 2023-07-04T21:27:56.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:27:56.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:28:26.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:28:26.713 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:28:26.751 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T21:28:26.749+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 662178) UID: 0 2023-07-04T21:28:26.751 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T21:28:26.749+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 662178) UID: 0 2023-07-04T21:28:26.751 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:28:26.750+0000 7f2bac08a700 -1 Fail to open '/proc/662178/cmdline' error = (2) No such file or directory 2023-07-04T21:28:26.752 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:28:26.750+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 662178) UID: 0 2023-07-04T21:28:26.753 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:28:26.751+0000 7f9b4664e700 -1 Fail to open '/proc/662178/cmdline' error = (2) No such file or directory 2023-07-04T21:28:26.753 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:28:26.751+0000 7f9b4664e700 -1 received signal: Hangup from (PID: 662178) UID: 0 2023-07-04T21:31:19.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:31:19.078 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:31:19.122 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T21:31:19.120+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 662225) UID: 0 2023-07-04T21:31:19.123 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T21:31:19.121+0000 7f7dfbfa6700 -1 received signal: Hangup from (PID: 662225) UID: 0 2023-07-04T21:31:19.123 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:31:19.121+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 662225) UID: 0 2023-07-04T21:31:19.125 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:31:19.121+0000 7f9b4664e700 -1 received signal: Hangup from (PID: 662225) UID: 0 2023-07-04T21:31:19.235 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T21:31:19.231+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148086) UID: 0 2023-07-04T21:31:19.235 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T21:31:19.231+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148086) UID: 0 2023-07-04T21:31:19.236 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T21:31:19.231+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148086) UID: 0 2023-07-04T21:31:19.236 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:31:19.232+0000 7fb432fbc700 -1 Fail to open '/proc/148086/cmdline' error = (2) No such file or directory 2023-07-04T21:31:19.236 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:31:19.232+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 148086) UID: 0 2023-07-04T21:34:15.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:34:15.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:34:46.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:34:46.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:35:16.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:35:16.227 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:35:46.291 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:35:46.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:36:16.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:36:16.428 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:36:46.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:36:46.589 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:37:16.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:37:16.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:37:47.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:37:47.109 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:38:17.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:38:17.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:38:47.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:38:47.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:39:17.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:39:17.468 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:39:47.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:39:47.557 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:40:17.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:40:17.710 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:40:47.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:40:47.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:41:17.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:41:17.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:41:48.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:41:48.012 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:42:18.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:42:18.309 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:42:48.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:42:48.396 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:43:18.529 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:43:18.533 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:43:48.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:43:48.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:44:18.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:44:18.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:44:48.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:44:48.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:45:19.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:45:19.144 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:45:49.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:45:49.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:46:19.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:46:19.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:46:19.387 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T21:46:19.384+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148746) UID: 0 2023-07-04T21:46:19.388 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T21:46:19.384+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148746) UID: 0 2023-07-04T21:46:19.389 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:46:19.385+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 148746) UID: 0 2023-07-04T21:46:19.389 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T21:46:19.384+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 148746) UID: 0 2023-07-04T21:49:15.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:49:15.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:49:45.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:49:45.666 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:50:15.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:50:15.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:50:45.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:50:45.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:51:16.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:51:16.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:51:46.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:51:46.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:52:16.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:52:16.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:52:46.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:52:46.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:53:16.799 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:53:16.802 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:53:17.102 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T21:53:17.101+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663343) UID: 0 2023-07-04T21:53:17.105 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T21:53:17.100+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663343) UID: 0 2023-07-04T21:53:17.105 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:53:17.102+0000 7f2bac08a700 -1 Fail to open '/proc/663343/cmdline' error = (2) No such file or directory 2023-07-04T21:53:17.106 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T21:53:17.102+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 663343) UID: 0 2023-07-04T21:53:17.106 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T21:53:17.100+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663343) UID: 0 2023-07-04T21:55:36.583 INFO:teuthology.orchestra.run.smithi077.stderr:error: Compressing program wrote following message to stderr when compressing log /var/log/ceph/ceph-osd.2.log.1: 2023-07-04T21:55:36.584 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: stdin: file size changed while zipping 2023-07-04T21:56:08.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:56:08.684 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:56:38.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:56:38.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:57:08.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:57:08.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:57:39.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:57:39.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:58:09.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:58:09.735 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:58:39.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:58:39.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:59:09.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:59:09.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:59:40.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:59:40.046 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T21:59:40.191 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T21:59:40.188+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149228) UID: 0 2023-07-04T21:59:40.192 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T21:59:40.188+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149228) UID: 0 2023-07-04T21:59:40.192 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T21:59:40.189+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149228) UID: 0 2023-07-04T21:59:40.193 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T21:59:40.189+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149228) UID: 0 2023-07-04T22:02:37.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:02:37.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:03:07.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:03:07.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:03:37.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:03:37.660 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:04:07.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:04:07.754 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:04:07.934 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:04:07.932+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663708) UID: 0 2023-07-04T22:04:07.934 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T22:04:07.932+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663708) UID: 0 2023-07-04T22:04:07.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:04:07.932+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663708) UID: 0 2023-07-04T22:04:07.937 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T22:04:07.932+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 663708) UID: 0 2023-07-04T22:06:54.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:06:54.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:07:24.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:07:24.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:07:54.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:07:54.344 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:08:24.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:08:24.464 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:08:54.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:08:54.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:09:25.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:09:25.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:09:55.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:09:55.377 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:10:25.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:10:25.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:10:55.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:10:55.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:11:25.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:11:25.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:11:55.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:11:55.941 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:12:26.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:12:26.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:12:56.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:12:56.358 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:13:26.445 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:13:26.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:13:56.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:13:56.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:13:56.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T22:13:56.679+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664114) UID: 0 2023-07-04T22:13:56.682 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:13:56.679+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664114) UID: 0 2023-07-04T22:13:56.682 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:13:56.679+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664114) UID: 0 2023-07-04T22:13:56.683 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T22:13:56.679+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664114) UID: 0 2023-07-04T22:16:41.771 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:16:41.774 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:16:41.874 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T22:16:41.873+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 149815) UID: 0 2023-07-04T22:16:41.875 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T22:16:41.873+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 149815) UID: 0 2023-07-04T22:19:50.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:19:50.706 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:20:21.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:20:21.070 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:20:51.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:20:51.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:21:21.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:21:21.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:21:51.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:21:51.343 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:22:21.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:22:21.476 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:22:21.544 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T22:22:21.543+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149991) UID: 0 2023-07-04T22:22:21.545 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T22:22:21.544+0000 7fb432fbc700 -1 Fail to open '/proc/149991/cmdline' error = (2) No such file or directory 2023-07-04T22:22:21.545 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T22:22:21.544+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 149991) UID: 0 2023-07-04T22:22:21.545 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T22:22:21.543+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 149991) UID: 0 2023-07-04T22:22:21.548 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T22:22:21.545+0000 7f218b0dd700 -1 Fail to open '/proc/149991/cmdline' error = (2) No such file or directory 2023-07-04T22:22:21.548 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T22:22:21.545+0000 7f218b0dd700 -1 received signal: Hangup from (PID: 149991) UID: 0 2023-07-04T22:25:18.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:25:18.117 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:25:18.198 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T22:25:18.195+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150042) UID: 0 2023-07-04T22:25:18.198 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T22:25:18.195+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150042) UID: 0 2023-07-04T22:25:18.199 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T22:25:18.195+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150042) UID: 0 2023-07-04T22:25:18.199 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T22:25:18.195+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150042) UID: 0 2023-07-04T22:28:21.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:28:21.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:28:51.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:28:51.730 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:29:21.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:29:21.810 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:29:52.107 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:29:52.110 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:30:22.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:30:22.428 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:30:52.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:30:52.742 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:31:22.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:31:22.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:31:52.904 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:31:52.910 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:32:23.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:32:23.021 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:32:53.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:32:53.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:32:53.203 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T22:32:53.201+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150327) UID: 0 2023-07-04T22:32:53.204 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T22:32:53.201+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150327) UID: 0 2023-07-04T22:32:53.204 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T22:32:53.201+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150327) UID: 0 2023-07-04T22:32:53.204 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T22:32:53.201+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 150327) UID: 0 2023-07-04T22:35:47.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:35:47.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:36:17.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:36:17.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:36:47.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:36:47.977 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:37:18.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:37:18.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:37:48.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:37:48.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:38:18.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:38:18.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:38:48.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:38:48.404 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:39:18.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:39:18.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:39:48.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:39:48.934 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:39:48.994 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T22:39:48.992+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664917) UID: 0 2023-07-04T22:39:48.994 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:39:48.992+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664917) UID: 0 2023-07-04T22:39:48.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T22:39:48.992+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664917) UID: 0 2023-07-04T22:39:48.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:39:48.993+0000 7f2bac08a700 -1 Fail to open '/proc/664917/cmdline' error = (2) No such file or directory 2023-07-04T22:39:48.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:39:48.993+0000 7f2bac08a700 -1 received signal: Hangup from (PID: 664917) UID: 0 2023-07-04T22:42:53.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:42:53.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:43:23.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:43:23.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:43:23.588 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:43:23.586+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664991) UID: 0 2023-07-04T22:43:23.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T22:43:23.586+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664991) UID: 0 2023-07-04T22:43:23.592 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T22:43:23.586+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664991) UID: 0 2023-07-04T22:43:23.593 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:43:23.586+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 664991) UID: 0 2023-07-04T22:46:15.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:46:15.587 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:46:45.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:46:45.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:47:15.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:47:15.930 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:47:46.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:47:46.058 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:48:16.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:48:16.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:48:46.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:48:46.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:49:16.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:49:16.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:49:46.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:49:46.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:50:16.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:50:16.794 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:50:46.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:50:46.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:51:17.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:51:17.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:51:47.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:51:47.362 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:52:17.463 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:52:17.466 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:52:47.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:52:47.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:53:17.872 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:53:17.874 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:53:47.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:53:48.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:54:18.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:54:18.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:54:48.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:54:48.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:55:18.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:55:18.362 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:55:18.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T22:55:18.416+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 665502) UID: 0 2023-07-04T22:55:18.418 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T22:55:18.416+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 665502) UID: 0 2023-07-04T22:55:18.418 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T22:55:18.416+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 665502) UID: 0 2023-07-04T22:55:18.420 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:55:18.417+0000 7f9b4664e700 -1 Fail to open '/proc/665502/cmdline' error = (2) No such file or directory 2023-07-04T22:55:18.420 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T22:55:18.417+0000 7f9b4664e700 -1 received signal: Hangup from (PID: 665502) UID: 0 2023-07-04T22:56:03.510 INFO:teuthology.orchestra.run.smithi077.stdout:generic/521 5545s 2023-07-04T22:56:03.605 INFO:teuthology.orchestra.run.smithi077.stdout:Ran: generic/521 2023-07-04T22:56:03.605 INFO:teuthology.orchestra.run.smithi077.stdout:Passed all 1 tests 2023-07-04T22:56:03.611 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T22:56:04.084 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-07-04T22:56:04.085 DEBUG:teuthology.orchestra.run.smithi077:> (cd /tmp/tmp.pA346HXQMhxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/522) 2023-07-04T22:56:05.121 INFO:teuthology.orchestra.run.smithi077.stdout:FSTYP -- ceph 2023-07-04T22:56:05.131 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM -- Linux/x86_64 smithi077 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 2023-07-04T22:56:05.132 INFO:teuthology.orchestra.run.smithi077.stdout:MKFS_OPTIONS -- 172.21.15.77:6789:/scratch 2023-07-04T22:56:05.132 INFO:teuthology.orchestra.run.smithi077.stdout:MOUNT_OPTIONS -- -o name=admin,secret=AQDvaaRk4ISsDhAAxWgUjdvU5lsU+Mq9QTwp7A== -o context=system_u:object_r:root_t:s0 172.21.15.77:6789:/scratch /tmp/tmp.nhbIsfbtq4scratch 2023-07-04T22:56:05.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-07-04T22:58:10.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:58:10.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:58:40.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:58:40.230 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:59:10.327 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:59:10.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:59:40.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T22:59:40.448 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:00:10.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:00:10.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:00:40.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:00:40.782 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:01:10.862 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:01:10.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:01:40.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:01:40.989 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:02:11.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:02:11.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:02:41.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:02:41.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:03:11.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:03:11.322 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:03:41.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:03:41.433 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:03:41.530 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T23:03:41.529+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 151688) UID: 0 2023-07-04T23:03:41.531 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T23:03:41.530+0000 7fb432fbc700 -1 Fail to open '/proc/151688/cmdline' error = (2) No such file or directory 2023-07-04T23:03:41.531 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T23:03:41.530+0000 7fb432fbc700 -1 received signal: Hangup from (PID: 151688) UID: 0 2023-07-04T23:03:41.532 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:03:41.530+0000 7f0e6e9ca700 -1 Fail to open '/proc/151688/cmdline' error = (2) No such file or directory 2023-07-04T23:03:41.532 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:03:41.530+0000 7f0e6e9ca700 -1 received signal: Hangup from (PID: 151688) UID: 0 2023-07-04T23:03:41.533 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T23:03:41.530+0000 7efd0b7fe700 -1 Fail to open '/proc/151688/cmdline' error = (2) No such file or directory 2023-07-04T23:03:41.533 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T23:03:41.530+0000 7efd0b7fe700 -1 received signal: Hangup from (PID: 151688) UID: 0 2023-07-04T23:06:40.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:06:40.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:06:40.933 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T23:06:40.932+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 151737) UID: 0 2023-07-04T23:06:40.933 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T23:06:40.932+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 151737) UID: 0 2023-07-04T23:06:40.934 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:06:40.932+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 151737) UID: 0 2023-07-04T23:06:40.934 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T23:06:40.933+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 151737) UID: 0 2023-07-04T23:06:41.063 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:06:41.061+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 666392) UID: 0 2023-07-04T23:06:41.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:06:41.061+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 666392) UID: 0 2023-07-04T23:06:41.063 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:06:41.061+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 666392) UID: 0 2023-07-04T23:06:41.064 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:06:41.061+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 666392) UID: 0 2023-07-04T23:09:48.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:09:48.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:10:18.767 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:10:18.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:10:48.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:10:48.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:11:18.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:11:18.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:11:49.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:11:49.261 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:12:19.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:12:19.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:12:49.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:12:49.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:13:19.916 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:13:19.918 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:13:50.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:13:50.373 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:14:20.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:14:20.443 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:14:20.505 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T23:14:20.504+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152015) UID: 0 2023-07-04T23:14:20.505 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T23:14:20.504+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152015) UID: 0 2023-07-04T23:14:20.506 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:14:20.505+0000 7f0e6e9ca700 -1 Fail to open '/proc/152015/cmdline' error = (2) No such file or directory 2023-07-04T23:14:20.506 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:14:20.505+0000 7f0e6e9ca700 -1 received signal: Hangup from (PID: 152015) UID: 0 2023-07-04T23:14:20.506 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T23:14:20.505+0000 7f218b0dd700 -1 Fail to open '/proc/152015/cmdline' error = (2) No such file or directory 2023-07-04T23:14:20.506 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T23:14:20.505+0000 7f218b0dd700 -1 received signal: Hangup from (PID: 152015) UID: 0 2023-07-04T23:17:26.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:17:26.074 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:17:56.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:17:56.527 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:18:26.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:18:26.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:18:56.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:18:56.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:19:27.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:19:27.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:19:57.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:19:57.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:20:27.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:20:27.367 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:20:57.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:20:57.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:21:27.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:21:27.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:21:57.907 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:21:57.909 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:22:28.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:22:28.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:22:58.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:22:58.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:23:28.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:23:28.474 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:23:58.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:23:58.564 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:23:58.634 INFO:tasks.ceph.osd.4.smithi082.stderr:2023-07-04T23:23:58.632+0000 7efd0b7fe700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152405) UID: 0 2023-07-04T23:23:58.634 INFO:tasks.ceph.osd.7.smithi082.stderr:2023-07-04T23:23:58.633+0000 7fb432fbc700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152405) UID: 0 2023-07-04T23:23:58.635 INFO:tasks.ceph.osd.6.smithi082.stderr:2023-07-04T23:23:58.632+0000 7f218b0dd700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152405) UID: 0 2023-07-04T23:23:58.639 INFO:tasks.ceph.osd.5.smithi082.stderr:2023-07-04T23:23:58.632+0000 7f0e6e9ca700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 152405) UID: 0 2023-07-04T23:26:57.482 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:26:57.485 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:27:27.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:27:27.779 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:27:58.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:27:58.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:28:28.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:28:28.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:28:58.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:28:58.393 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:29:28.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:29:28.476 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:29:58.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:29:58.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:29:58.671 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:29:58.670+0000 7f4f6a8d1700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 667252) UID: 0 2023-07-04T23:29:58.672 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:29:58.670+0000 7f7dfbfa6700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 667252) UID: 0 2023-07-04T23:29:58.672 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:29:58.670+0000 7f9b4664e700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 667252) UID: 0 2023-07-04T23:29:58.673 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:29:58.670+0000 7f2bac08a700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 667252) UID: 0 2023-07-04T23:32:45.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:32:45.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:33:15.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:33:15.688 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:33:15.759 INFO:tasks.ceph.mds.d.smithi082.stderr:2023-07-04T23:33:15.757+0000 7fa6fcde5700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 152677) UID: 0 2023-07-04T23:33:15.759 INFO:tasks.ceph.mds.b.smithi082.stderr:2023-07-04T23:33:15.757+0000 7fb0bf5a3700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 152677) UID: 0 2023-07-04T23:36:26.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:36:26.597 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:36:35.683 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:35.682+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:35.683 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:35.682+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:35.683 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:35.682+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:36.200 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:36.199+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:36.200 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:36.199+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:36.200 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:36.199+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:36.580 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:36.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:36.581 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:36.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:36.581 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:36.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:36.581 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:36.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:36.706 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:36.705+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:36.706 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:36.705+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:36.707 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:36.705+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:36.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:36.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:36.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:36.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:36.981 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:36.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:37.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:37.198+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:37.199 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:37.198+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:37.199 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:37.198+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:37.571+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:37.571+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:37.571+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:37.571+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:37.658 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:37.657+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:37.658 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:37.657+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:37.659 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:37.657+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:37.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:37.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:37.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:37.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:37.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:37.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:38.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:38.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:38.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:38.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:38.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:38.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:38.585 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:38.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:38.585 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:38.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:38.586 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:38.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:38.586 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:38.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:38.677 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:38.677+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:38.678 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:38.677+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:38.678 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:38.677+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:38.970 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:38.969+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:38.970 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:38.969+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:38.970 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:38.969+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:39.204 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:39.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:39.204 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:39.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:39.205 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:39.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:39.623 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:39.622+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:39.623 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:39.622+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:39.623 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:39.622+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:39.624 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:39.622+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:39.646 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:39.646+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:39.647 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:39.646+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:39.647 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:39.646+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:39.924 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:39.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:39.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:39.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:39.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:39.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:40.169 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:40.169+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:40.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:40.169+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:40.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:40.169+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:40.625 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:40.624+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:40.625 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:40.624+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:40.625 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:40.624+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:40.658 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:40.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:40.658 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:40.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:40.659 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:40.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:40.659 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:40.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:40.889 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:40.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:40.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:40.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:40.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:40.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:41.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:41.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:41.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:41.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:41.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:41.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:41.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:41.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:41.651 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:41.650+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:41.652 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:41.650+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:41.652 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:41.650+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:41.670 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:41.670+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:41.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:41.670+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:41.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:41.670+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:41.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:41.670+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:41.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:41.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:41.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:41.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:41.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:41.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:41.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:41.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:42.139 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:42.138+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:42.139 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:42.138+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:42.139 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:42.138+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:42.139 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:42.138+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:42.653 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:42.653+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:42.654 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:42.653+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:42.654 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:42.653+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:42.689 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:42.689+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:42.690 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:42.689+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:42.690 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:42.689+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:42.690 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:42.689+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:42.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:42.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:42.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:42.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:42.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:42.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:42.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:42.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:43.108 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:43.108+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:43.109 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:43.108+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:43.109 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:43.108+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:43.109 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:43.108+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:43.626 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:43.625+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:43.626 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:43.625+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:43.626 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:43.625+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:43.640 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:43.640+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:43.640 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:43.640+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:43.640 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:43.640+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:43.640 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:43.640+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:43.848 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:43.847+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:43.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:43.847+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:43.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:43.847+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:43.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:43.847+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:44.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:44.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:44.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:44.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:44.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:44.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:44.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:44.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:44.628 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:44.628+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:44.629 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:44.628+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:44.629 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:44.628+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:44.629 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:44.628+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:44.658 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:44.658+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:44.658 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:44.658+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:44.659 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:44.658+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:44.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:44.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:44.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:44.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:44.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:44.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:44.876 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:44.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:45.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:45.033+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:45.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:45.033+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:45.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:45.033+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:45.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:45.033+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:45.652 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:45.651+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:45.652 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:45.651+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:45.652 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:45.651+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:45.655 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:45.654+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:45.655 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:45.654+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:45.656 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:45.654+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:45.656 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:45.654+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:45.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:45.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:45.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:45.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:45.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:45.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:45.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:45.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:46.048 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:46.047+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:46.048 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:46.047+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:46.048 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:46.047+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:46.048 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:46.047+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:46.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:46.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:46.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:46.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:46.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:46.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:46.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:46.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:46.679 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:46.678+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:46.679 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:46.678+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:46.679 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:46.678+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:46.679 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:46.678+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:46.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:46.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:46.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:46.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:47.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:47.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:47.009 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:47.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:47.009 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:47.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:47.009 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:47.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:47.624 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:47.623+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:47.624 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:47.623+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:47.625 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:47.623+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:47.625 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:47.623+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:47.637 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:47.636+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:47.637 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:47.636+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:47.637 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:47.636+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:47.638 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:47.636+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:47.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:47.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:47.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:47.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:47.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:47.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:47.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:47.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:48.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:48.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:48.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:48.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:48.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:48.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:48.059 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:48.058+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:48.631 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:48.631+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:48.631 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:48.631+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:48.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:48.631+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:48.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:48.631+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:48.668 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:48.667+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:48.668 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:48.667+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:48.668 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:48.667+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:48.669 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:48.667+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:48.903 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:48.902+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:48.903 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:48.902+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:48.903 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:48.902+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:48.903 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:48.902+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:49.084 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:49.084+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:49.084 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:49.084+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:49.084 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:49.084+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:49.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:49.084+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:49.648 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:49.648+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:49.649 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:49.648+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:49.649 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:49.648+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:49.649 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:49.648+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:49.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:49.671+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:49.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:49.671+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:49.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:49.671+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:49.671 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:49.671+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:49.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:49.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:49.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:49.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:49.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:49.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:49.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:49.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:50.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:50.120+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:50.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:50.120+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:50.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:50.120+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:50.121 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:50.120+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:50.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:50.653+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:50.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:50.653+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:50.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:50.653+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:50.654 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:50.653+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:50.698 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:50.697+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:50.698 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:50.697+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:50.698 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:50.697+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:50.698 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:50.697+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:50.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:50.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:50.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:50.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:50.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:50.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:50.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:50.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:51.140 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:51.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:51.141 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:51.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:51.141 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:51.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:51.141 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:51.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:51.673 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:51.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:51.673 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:51.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:51.673 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:51.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:51.674 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:51.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:51.747 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:51.747+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:51.748 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:51.747+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:51.748 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:51.747+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:51.748 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:51.747+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:51.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:51.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:51.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:51.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:51.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:51.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:51.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:51.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:52.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:52.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:52.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:52.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:52.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:52.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:52.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:52.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:52.723 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:52.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:52.723 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:52.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:52.723 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:52.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:52.723 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:52.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:52.752 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:52.752+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:52.752 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:52.752+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:52.752 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:52.752+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:52.753 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:52.752+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:52.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:52.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:52.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:52.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:52.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:52.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:52.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:52.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:53.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:53.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:53.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:53.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:53.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:53.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:53.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:53.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:53.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:53.760+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:53.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:53.760+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:53.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:53.760+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:53.761 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:53.760+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:53.789 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:53.789+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:53.789 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:53.789+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:53.789 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:53.789+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:53.790 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:53.789+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:53.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:53.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:53.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:53.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:53.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:53.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:53.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:53.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:54.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:54.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:54.181 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:54.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:54.182 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:54.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:54.182 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:54.180+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:54.719 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:54.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:54.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:54.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:54.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:54.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:54.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:54.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:54.770 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:54.768+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:54.770 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:54.768+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:54.770 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:54.768+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:54.770 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:54.768+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:54.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:54.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:54.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:54.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:54.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:54.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:54.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:54.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:55.195 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:55.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:55.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:55.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:55.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:55.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:55.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:55.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:55.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:55.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:55.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:55.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:55.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:55.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:55.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:55.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:55.804 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:55.803+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:55.805 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:55.803+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:55.805 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:55.803+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:55.805 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:55.803+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:55.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:55.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:55.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:55.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:55.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:55.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:55.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:55.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:56.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:56.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:56.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:56.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:56.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:56.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:56.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:56.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:56.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:56.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:56.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:56.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:56.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:56.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:56.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:56.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:56.760 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:56.759+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:56.760 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:56.759+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:56.760 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:56.759+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:56.760 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:56.759+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:56.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:56.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:56.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:56.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:56.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:56.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:56.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:56.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:57.243 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:57.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:57.243 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:57.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:57.243 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:57.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:57.243 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:57.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:57.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:57.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:57.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:57.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:57.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:57.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:57.686 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:57.684+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:57.739 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:57.738+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:57.740 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:57.738+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:57.740 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:57.738+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:57.740 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:57.738+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:57.842 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:57.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:57.842 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:57.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:57.842 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:57.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:57.842 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:57.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:58.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:58.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:58.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:58.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:58.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:58.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:58.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:58.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:58.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:58.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:58.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:58.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:58.654 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:58.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:58.654 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:58.652+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:58.743 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:58.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:58.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:58.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:58.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:58.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:58.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:58.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:58.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:58.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:58.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:58.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:58.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:58.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:58.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:58.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:59.166 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:59.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:59.167 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:59.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:36:59.167 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:59.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:59.167 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:36:59.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:36:59.611 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:59.610+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:59.611 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:59.610+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:59.611 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:59.610+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:59.611 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:36:59.610+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:36:59.724 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:59.723+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:59.724 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:59.723+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:59.724 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:59.723+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:36:59.725 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:36:59.723+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:36:59.876 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:59.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:59.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:59.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:36:59.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:59.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:36:59.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:36:59.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:00.169 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:00.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:00.169 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:00.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:00.169 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:00.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:00.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:00.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:00.635 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:00.634+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:00.635 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:00.634+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:00.636 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:00.634+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:00.636 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:00.634+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:00.691 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:00.690+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:00.691 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:00.690+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:00.692 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:00.690+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:00.692 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:00.690+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:00.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:00.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:00.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:00.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:00.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:00.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:00.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:00.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:01.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:01.218+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:01.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:01.218+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:01.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:01.218+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:01.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:01.218+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:01.609 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:01.608+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:01.609 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:01.608+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:01.609 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:01.608+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:01.610 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:01.608+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:01.670 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:01.669+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:01.670 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:01.669+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:01.670 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:01.669+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:01.671 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:01.669+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:01.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:01.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:01.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:01.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:01.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:01.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:01.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:01.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:02.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:02.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:02.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:02.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:02.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:02.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:02.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:02.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:02.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:02.568+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:02.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:02.568+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:02.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:02.568+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:02.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:02.568+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:02.631 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:02.630+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:02.632 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:02.630+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:02.632 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:02.630+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:02.632 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:02.630+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:02.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:02.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:02.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:02.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:02.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:02.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:02.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:02.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:03.227 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:03.226+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:03.227 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:03.226+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:03.228 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:03.226+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:03.228 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:03.226+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:03.581 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:03.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:03.582 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:03.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:03.582 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:03.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:03.582 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:03.580+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:03.635 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:03.634+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:03.636 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:03.634+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:03.636 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:03.634+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:03.636 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:03.634+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:03.876 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:03.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:03.876 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:03.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:03.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:03.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:03.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:03.875+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:04.207 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:04.206+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:04.208 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:04.206+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:04.208 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:04.206+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:04.208 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:04.206+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:04.561 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:04.560+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:04.561 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:04.560+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:04.561 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:04.560+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:04.561 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:04.560+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:04.673 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:04.672+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:04.673 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:04.672+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:04.673 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:04.672+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:04.673 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:04.672+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:04.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:04.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:04.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:04.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:04.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:04.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:04.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:04.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:05.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:05.164+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:05.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:05.164+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:05.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:05.164+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:05.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:05.164+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:05.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:05.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:05.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:05.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:05.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:05.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:05.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:05.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:05.708 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:05.707+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:05.708 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:05.707+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:05.708 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:05.707+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:05.708 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:05.707+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:05.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:05.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:05.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:05.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:05.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:05.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:05.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:05.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:06.141 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:06.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:06.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:06.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:06.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:06.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:06.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:06.140+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:06.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:06.569+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:06.570 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:06.569+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:06.571 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:06.569+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:06.571 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:06.569+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:06.743 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:06.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:06.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:06.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:06.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:06.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:06.744 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:06.742+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:06.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:06.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:06.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:06.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:06.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:06.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:06.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:06.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:07.174 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:07.173+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:07.174 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:07.173+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:07.174 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:07.173+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:07.174 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:07.173+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:07.551 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:07.550+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:07.552 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:07.550+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:07.552 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:07.550+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:07.552 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:07.550+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:07.722 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:07.722+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:07.723 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:07.722+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:07.723 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:07.722+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:07.723 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:07.722+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:07.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:07.983+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:07.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:07.983+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:07.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:07.983+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:07.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:07.983+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:08.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:08.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:08.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:08.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:08.159 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:08.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:08.160 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:08.158+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:08.508 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:08.507+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:08.508 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:08.507+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:08.508 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:08.507+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:08.508 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:08.507+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:08.767 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:08.766+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:08.768 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:08.766+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:08.768 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:08.766+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:08.768 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:08.766+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:08.954 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:08.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:08.954 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:08.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:08.955 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:08.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:08.955 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:08.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:09.143 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:09.142+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:09.143 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:09.142+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:09.144 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:09.142+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:09.144 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:09.142+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:09.545 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:09.544+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:09.545 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:09.544+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:09.545 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:09.544+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:09.546 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:09.544+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:09.759 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:09.758+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:09.759 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:09.758+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:09.759 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:09.758+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:09.759 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:09.758+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:09.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:09.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:09.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:09.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:09.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:09.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:09.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:09.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:10.098 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:10.097+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:10.098 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:10.097+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:10.098 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:10.097+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:10.098 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:10.097+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:10.533 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:10.532+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:10.533 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:10.532+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:10.533 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:10.532+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:10.533 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:10.532+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:10.772 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:10.771+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:10.772 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:10.771+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:10.773 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:10.771+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:10.773 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:10.771+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:10.974 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:10.973+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:10.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:10.973+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:10.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:10.973+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:10.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:10.973+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:11.091 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:11.090+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:11.092 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:11.090+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:11.092 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:11.090+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:11.092 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:11.090+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:11.549 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:11.548+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:11.549 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:11.548+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:11.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:11.548+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:11.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:11.548+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:11.822 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:11.820+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:11.822 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:11.820+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:11.822 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:11.820+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:11.822 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:11.820+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:12.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:12.009+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:12.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:12.009+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:12.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:12.009+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:12.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:12.009+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:12.061 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:12.060+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:12.062 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:12.060+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:12.062 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:12.060+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:12.062 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:12.060+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:12.585 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:12.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:12.585 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:12.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:12.586 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:12.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:12.586 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:12.584+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:12.871 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:12.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:12.871 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:12.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:12.872 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:12.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:12.872 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:12.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:13.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:13.006+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:13.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:13.006+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:13.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:13.006+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:13.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:13.006+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:13.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:13.035+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:13.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:13.035+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:13.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:13.035+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:13.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:13.035+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:13.604 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:13.602+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:13.604 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:13.602+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:13.604 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:13.602+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:13.604 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:13.602+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:13.865 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:13.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:13.865 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:13.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:13.865 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:13.864+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:13.865 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:13.864+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:14.010 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.009+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:14.010 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.009+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:14.010 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.009+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:14.010 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.009+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:14.021 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.020+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.020+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.020+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:14.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.020+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.613 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:14.612+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:14.613 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:14.612+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:14.613 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:14.612+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:14.614 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:14.612+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:14.914 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:14.913+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:14.915 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:14.913+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:14.915 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:14.913+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:14.915 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:14.913+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:14.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:14.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:14.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:14.993 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.992+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:14.993 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.992+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:14.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.992+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:14.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:14.992+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:15.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:15.633+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:15.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:15.633+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:15.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:15.633+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:15.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:15.633+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:15.927 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:15.926+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:15.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:15.926+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:15.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:15.926+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:15.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:15.926+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:15.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:15.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:15.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:15.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:15.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:15.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:15.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:15.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:15.996 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:15.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:15.996 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:15.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:15.996 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:15.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:15.996 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:15.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:16.641 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:16.641+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:16.642 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:16.641+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:16.642 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:16.641+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:16.642 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:16.641+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:16.883 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:16.882+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:16.884 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:16.882+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:16.884 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:16.882+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:16.884 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:16.882+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:16.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:16.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:16.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:16.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:16.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:16.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:16.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:16.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:17.013 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.012+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:17.013 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.012+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:17.013 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.012+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:17.013 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.012+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:17.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:17.677+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:17.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:17.677+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:17.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:17.677+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:17.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:17.677+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:17.912 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:17.911+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:17.912 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:17.911+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:17.913 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:17.911+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:17.913 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:17.911+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:17.963 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:17.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:17.963 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:17.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:17.964 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:17.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:17.964 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:17.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:17.982 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.981+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:17.982 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.981+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:17.982 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.981+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:17.982 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:17.981+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:18.665 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:18.664+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:18.666 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:18.664+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:18.666 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:18.664+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:18.666 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:18.664+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:18.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:18.884+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:18.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:18.884+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:18.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:18.884+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:18.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:18.884+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:19.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.002+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.002+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.002+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:19.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.002+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:19.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:19.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:19.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:19.688 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:19.688+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:19.689 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:19.688+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:19.689 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:19.688+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:19.689 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:19.688+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:19.838 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:19.838+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:19.839 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:19.838+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:19.839 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:19.838+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:19.839 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:19.838+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:19.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:19.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:19.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:19.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.996+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:19.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.996+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:19.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.996+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:19.998 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:19.996+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:20.675 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:20.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:20.675 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:20.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:20.675 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:20.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:20.675 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:20.673+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:20.868 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:20.867+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:20.868 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:20.867+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:20.868 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:20.867+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:20.868 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:20.867+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:20.965 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:20.964+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:20.965 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:20.964+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:20.965 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:20.964+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:20.965 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:20.964+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:20.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:20.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:20.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:20.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:20.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:20.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:20.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:20.979+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:21.658 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:21.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:21.659 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:21.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:21.659 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:21.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:21.659 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:21.658+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:21.910 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:21.909+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:21.910 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:21.909+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:21.910 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:21.909+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:21.911 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:21.909+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:21.986 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:21.985+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:21.986 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:21.985+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:21.986 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:21.985+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:21.986 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:21.985+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:21.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:21.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:21.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:21.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:21.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:21.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:21.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:21.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:22.708 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:22.707+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:22.708 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:22.707+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:22.708 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:22.707+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:22.709 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:22.707+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:22.878 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:22.877+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:22.878 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:22.877+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:22.878 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:22.877+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:22.878 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:22.877+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:23.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:23.033+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:23.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:23.033+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:23.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:23.033+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:23.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:23.033+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:23.667 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:23.666+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:23.667 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:23.666+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:23.668 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:23.666+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:23.668 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:23.666+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:23.925 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:23.925+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:23.926 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:23.925+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:23.926 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:23.925+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:23.926 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:23.925+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:24.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:24.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:24.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:24.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:24.014 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:24.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:24.014 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:24.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:24.022 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:24.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:24.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:24.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:24.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:24.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:24.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:24.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:24.662 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:24.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:24.662 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:24.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:24.662 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:24.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:24.663 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:24.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:24.966 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:24.964+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:24.966 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:24.964+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:24.966 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:24.964+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:24.966 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:24.964+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:25.056 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:25.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:25.057 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:25.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:25.057 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:25.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:25.057 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:25.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:25.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:25.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:25.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:25.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:25.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:25.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:25.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:25.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:25.687 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:25.687+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:25.688 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:25.687+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:25.688 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:25.687+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:25.688 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:25.687+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:26.014 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.013+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.014 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.013+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.015 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.013+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.015 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.013+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:26.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:26.051+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:26.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:26.051+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:26.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:26.051+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:26.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:26.051+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:26.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:26.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:26.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:26.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:26.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:26.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:26.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:26.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:26.716 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:26.716+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:26.716 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:26.716+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:26.717 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:26.716+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:26.717 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:26.716+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:26.985 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.984+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.985 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.984+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.985 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.984+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:26.985 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:26.984+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:27.029 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:27.029+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:27.030 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:27.029+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:27.030 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:27.029+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:27.030 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:27.029+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:27.032 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.031+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:27.032 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.031+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:27.032 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.031+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:27.032 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.031+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:27.719 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:27.719+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:27.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:27.719+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:27.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:27.719+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:27.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:27.719+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:27.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:27.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:27.998 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:27.998 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:27.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:28.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:28.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:28.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:28.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:28.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:28.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:28.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:28.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:28.077 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:28.077+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:28.077 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:28.077+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:28.078 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:28.077+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:28.078 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:28.077+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:28.683 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:28.683+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:28.683 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:28.683+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:28.684 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:28.683+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:28.684 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:28.683+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:28.966 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:28.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:28.967 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:28.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:28.967 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:28.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:28.967 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:28.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:29.074 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:29.074+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:29.074 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:29.074+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:29.074 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:29.074+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:29.075 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:29.074+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:29.127 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:29.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:29.127 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:29.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:29.128 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:29.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:29.128 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:29.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:29.729 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:29.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:29.729 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:29.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:29.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:29.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:29.730 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:29.728+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:29.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:29.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:29.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:29.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:29.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:29.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:29.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:29.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:30.087 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:30.086+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:30.087 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:30.086+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:30.087 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:30.086+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:30.088 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:30.086+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:30.088 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:30.087+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:30.088 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:30.087+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:30.088 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:30.087+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:30.088 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:30.087+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:30.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:30.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:30.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:30.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:30.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:30.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:30.681 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:30.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:30.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:30.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:30.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:30.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:30.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:30.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:30.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:30.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:31.085 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:31.084+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:31.085 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:31.084+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:31.085 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:31.084+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:31.085 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:31.084+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:31.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:31.130+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:31.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:31.130+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:31.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:31.130+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:31.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:31.130+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:31.703 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:31.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:31.703 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:31.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:31.703 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:31.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:31.704 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:31.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:32.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:32.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:32.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:32.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:32.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:32.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:32.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:32.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:32.049 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:32.047+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:32.049 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:32.047+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:32.049 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:32.047+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:32.049 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:32.047+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:32.096 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:32.096+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:32.096 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:32.096+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:32.096 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:32.096+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:32.097 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:32.096+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:32.663 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:32.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:32.663 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:32.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:32.663 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:32.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:32.664 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:32.662+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:33.023 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:33.023+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:33.024 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:33.023+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:33.024 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:33.023+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:33.024 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:33.023+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:33.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:33.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:33.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:33.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:33.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:33.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:33.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:33.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:33.145 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:33.145+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:33.145 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:33.145+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:33.145 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:33.145+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:33.146 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:33.145+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:33.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:33.678+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:33.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:33.678+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:33.679 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:33.678+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:33.679 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:33.678+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:34.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.033+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.034 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.033+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.034 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.033+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.034 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.033+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:34.087 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:34.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:34.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:34.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:34.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:34.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:34.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:34.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:34.184 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:34.183+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:34.184 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:34.183+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:34.184 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:34.183+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:34.184 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:34.183+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:34.631 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:34.630+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:34.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:34.630+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:34.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:34.630+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:34.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:34.630+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:34.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.988+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.989 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.988+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.989 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.988+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:34.989 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:34.988+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:35.129 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:35.129+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:35.129 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:35.129+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:35.129 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:35.129+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:35.130 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:35.129+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:35.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:35.161+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:35.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:35.161+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:35.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:35.161+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:35.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:35.161+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:35.606 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:35.606+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:35.607 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:35.606+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:35.607 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:35.606+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:35.607 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:35.606+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:35.983 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:35.983+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:35.983 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:35.983+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:35.984 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:35.983+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:35.984 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:35.983+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:36.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:36.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:36.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:36.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:36.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:36.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:36.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:36.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:36.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:36.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:36.165 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:36.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:36.166 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:36.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:36.166 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:36.165+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:36.565 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:36.565+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:36.566 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:36.565+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:36.566 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:36.565+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:36.566 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:36.565+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:36.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:36.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:36.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:36.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:36.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:36.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:36.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:36.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:37.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:37.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:37.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:37.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:37.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:37.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:37.094 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:37.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:37.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:37.141+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:37.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:37.141+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:37.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:37.141+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:37.142 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:37.141+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:37.559 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:37.558+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:37.559 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:37.558+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:37.559 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:37.558+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:37.559 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:37.558+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:37.941 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:37.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:37.942 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:37.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:37.942 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:37.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:37.942 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:37.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:38.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:38.127+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:38.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:38.127+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:38.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:38.127+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:38.128 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:38.127+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:38.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:38.162+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:38.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:38.162+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:38.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:38.162+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:38.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:38.162+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:38.548 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:38.547+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:38.548 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:38.547+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:38.548 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:38.547+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:38.548 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:38.547+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:38.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:38.947+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:38.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:38.947+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:38.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:38.947+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:38.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:38.947+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:39.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:39.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:39.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:39.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:39.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:39.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:39.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:39.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:39.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:39.203+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:39.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:39.203+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:39.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:39.203+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:39.204 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:39.203+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:39.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:39.516+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:39.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:39.516+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:39.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:39.516+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:39.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:39.516+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:39.996 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:39.995+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:39.996 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:39.995+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:39.996 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:39.995+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:39.996 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:39.995+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:40.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:40.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:40.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:40.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:40.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:40.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:40.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:40.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:40.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:40.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:40.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:40.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:40.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:40.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:40.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:40.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:40.527 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:40.527+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:40.527 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:40.527+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:40.527 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:40.527+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:40.528 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:40.527+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:41.016 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.016 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.016 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.016 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:41.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:41.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:41.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:41.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:41.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:41.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:41.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:41.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:41.186 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:41.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:41.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:41.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:41.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:41.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:41.187 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:41.186+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:41.520 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:41.519+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:41.520 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:41.519+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:41.521 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:41.519+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:41.521 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:41.519+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:41.998 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.997+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.998 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.997+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.998 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.997+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:41.998 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:41.997+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:42.081 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:42.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:42.081 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:42.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:42.081 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:42.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:42.082 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:42.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:42.205 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:42.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:42.206 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:42.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:42.206 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:42.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:42.206 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:42.204+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:42.505 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:42.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:42.505 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:42.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:42.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:42.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:42.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:42.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:43.003 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:43.002+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:43.004 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:43.002+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:43.004 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:43.002+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:43.004 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:43.002+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:43.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:43.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:43.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:43.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:43.064 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:43.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:43.064 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:43.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:43.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:43.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:43.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:43.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:43.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:43.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:43.170 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:43.168+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:43.493 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:43.492+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:43.494 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:43.492+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:43.494 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:43.492+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:43.494 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:43.492+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:44.020 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:44.019+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:44.021 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:44.019+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:44.021 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:44.019+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:44.021 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:44.019+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:44.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:44.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:44.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:44.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:44.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:44.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:44.094 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:44.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:44.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:44.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:44.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:44.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:44.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:44.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:44.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:44.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:44.460 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:44.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:44.460 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:44.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:44.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:44.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:44.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:44.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:45.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:45.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:45.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:45.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:45.048 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:45.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:45.048 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:45.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:45.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:45.085+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:45.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:45.085+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:45.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:45.085+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:45.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:45.085+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:45.233 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:45.232+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:45.233 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:45.232+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:45.233 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:45.232+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:45.233 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:45.232+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:45.426 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:45.425+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:45.426 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:45.425+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:45.426 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:45.425+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:45.427 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:45.425+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:46.012 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.010+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:46.012 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.010+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:46.012 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.010+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:46.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.010+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:46.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:46.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:46.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:46.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:46.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:46.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:46.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:46.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:46.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:46.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:46.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:46.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:46.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:46.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:46.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:46.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:46.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:46.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:46.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:46.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:46.475 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:46.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:46.475 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:46.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:47.000 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.999+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:47.000 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.999+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:47.001 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.999+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:47.001 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:46.999+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:47.106 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:47.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:47.106 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:47.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:47.106 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:47.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:47.106 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:47.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:47.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:47.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:47.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:47.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:47.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:47.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:47.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:47.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:47.509 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:47.508+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:47.509 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:47.508+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:47.509 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:47.508+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:47.510 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:47.508+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:48.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:48.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:48.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:48.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:48.033 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:48.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:48.034 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:48.032+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:48.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:48.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:48.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:48.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:48.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:48.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:48.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:48.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:48.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:48.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:48.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:48.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:48.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:48.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:48.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:48.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:48.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:48.472+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:48.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:48.472+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:48.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:48.472+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:48.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:48.472+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:49.028 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:49.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:49.028 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:49.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:49.028 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:49.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:49.028 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:49.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:49.057 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:49.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:49.058 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:49.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:49.058 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:49.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:49.058 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:49.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:49.173 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:49.172+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:49.173 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:49.172+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:49.173 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:49.172+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:49.173 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:49.172+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:49.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:49.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:49.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:49.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:49.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:49.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:49.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:49.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:50.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:50.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:50.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:50.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:50.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:50.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:50.053 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:50.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:50.072 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:50.071+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:50.072 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:50.071+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:50.072 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:50.071+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:50.072 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:50.071+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:50.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:50.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:50.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:50.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:50.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:50.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:50.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:50.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:50.477 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:50.475+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:50.477 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:50.475+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:50.477 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:50.475+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:50.477 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:50.475+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:51.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:51.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:51.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:51.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:51.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:51.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:51.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:51.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:51.092 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:51.091+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:51.092 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:51.091+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:51.092 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:51.091+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:51.092 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:51.091+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:51.207 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:51.205+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:51.207 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:51.205+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:51.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:51.205+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:51.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:51.205+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:51.510 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:51.509+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:51.511 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:51.509+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:51.511 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:51.509+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:51.511 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:51.509+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:52.077 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:52.076+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:52.077 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:52.076+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:52.077 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:52.076+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:52.078 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:52.076+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:52.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:52.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:52.115 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:52.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:52.115 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:52.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:52.115 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:52.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:52.222 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:52.221+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:52.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:52.221+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:52.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:52.221+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:52.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:52.221+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:52.462 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:52.461+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:52.462 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:52.461+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:52.462 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:52.461+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:52.463 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:52.461+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:53.035 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:53.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:53.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:53.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:53.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:53.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:53.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:53.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:53.149 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:53.148+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:53.149 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:53.148+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:53.149 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:53.148+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:53.149 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:53.148+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:53.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:53.248+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:53.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:53.248+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:53.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:53.248+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:53.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:53.248+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:53.511 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:53.510+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:53.512 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:53.510+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:53.512 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:53.510+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:53.512 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:53.510+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:54.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:54.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:54.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:54.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:54.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:54.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:54.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:54.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:54.159 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:54.158+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:54.159 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:54.158+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:54.159 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:54.158+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:54.160 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:54.158+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:54.248 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:54.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:54.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:54.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:54.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:54.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:54.249 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:54.247+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:54.495 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:54.494+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:54.495 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:54.494+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:54.496 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:54.494+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:54.496 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:54.494+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:55.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:55.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:55.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:55.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:55.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:55.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:55.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:55.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:55.168 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:55.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:55.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:55.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:55.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:55.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:55.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:55.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:55.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:55.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:55.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:55.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:55.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:55.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:55.203 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:55.202+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:55.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:55.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:55.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:55.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:55.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:55.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:55.461 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:55.459+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:56.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:56.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:56.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:56.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:56.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:56.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:56.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:56.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:56.152 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:56.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:56.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:56.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:56.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:56.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:56.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:56.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:56.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:56.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:56.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:56.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:56.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:56.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:56.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:56.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:56.485 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:56.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:56.485 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:56.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:56.486 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:56.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:56.486 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:56.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:57.140 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:57.139+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:57.141 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:57.139+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:57.141 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:57.139+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:57.141 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:57.139+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:57.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:57.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:57.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:57.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:57.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:57.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:57.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:57.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:57.241 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:57.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:57.241 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:57.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:57.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:57.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:57.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:57.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:57.458 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:57.457+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:57.458 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:57.457+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:57.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:57.457+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:57.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:57.457+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:58.095 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:58.094+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:58.095 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:58.094+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:58.095 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:58.094+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:58.096 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:58.094+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:58.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:58.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:58.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:58.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:58.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:58.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:58.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:58.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:58.279 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:58.278+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:58.279 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:58.278+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:58.280 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:58.278+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:58.280 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:58.278+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:58.478 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:58.477+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:58.479 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:58.477+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:58.479 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:58.477+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:58.479 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:58.477+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:59.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:59.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:59.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:59.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:59.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:59.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:37:59.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:37:59.055+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:37:59.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:59.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:59.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:59.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:59.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:59.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:37:59.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:37:59.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:37:59.318 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:59.317+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:59.318 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:59.317+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:37:59.318 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:59.317+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:59.318 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:37:59.317+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:37:59.453 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:59.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:59.453 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:59.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:59.453 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:59.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:37:59.453 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:37:59.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:00.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:00.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:00.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:00.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:00.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:00.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:00.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:00.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:00.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:00.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:00.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:00.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:00.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:00.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:00.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:00.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:00.288 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:00.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:00.288 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:00.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:00.288 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:00.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:00.288 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:00.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:00.470 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:00.469+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:00.470 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:00.469+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:00.471 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:00.469+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:00.471 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:00.469+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:01.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:01.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:01.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:01.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:01.023 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:01.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:01.023 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:01.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:01.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:01.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:01.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:01.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:01.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:01.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:01.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:01.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:01.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:01.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:01.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:01.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:01.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:01.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:01.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:01.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:01.447 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:01.446+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:01.447 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:01.446+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:01.447 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:01.446+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:01.448 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:01.446+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:02.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:02.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:02.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:02.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:02.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:02.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:02.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:02.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:02.233 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:02.232+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:02.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:02.232+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:02.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:02.232+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:02.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:02.232+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:02.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:02.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:02.374 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:02.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:02.374 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:02.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:02.374 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:02.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:02.483 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:02.482+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:02.483 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:02.482+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:02.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:02.482+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:02.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:02.482+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:03.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:03.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:03.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:03.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:03.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:03.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:03.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:03.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:03.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:03.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:03.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:03.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:03.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:03.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:03.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:03.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:03.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:03.331+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:03.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:03.331+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:03.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:03.331+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:03.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:03.331+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:03.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:03.515+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:03.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:03.515+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:03.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:03.515+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:03.516 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:03.515+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:04.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:04.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:04.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:04.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:04.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:04.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:04.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:04.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:04.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:04.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:04.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:04.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:04.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:04.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:04.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:04.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:04.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:04.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:04.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:04.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:04.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:04.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:04.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:04.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:04.525 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:04.523+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:04.525 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:04.523+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:04.525 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:04.523+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:04.525 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:04.523+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:05.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:05.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:05.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:05.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:05.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:05.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:05.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:05.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:05.190 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:05.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:05.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:05.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:05.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:05.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:05.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:05.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:05.328 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:05.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:05.329 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:05.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:05.329 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:05.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:05.329 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:05.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:05.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:05.549+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:05.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:05.549+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:05.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:05.549+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:05.550 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:05.549+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:06.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:06.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:06.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:06.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:06.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:06.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:06.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:06.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:06.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:06.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:06.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:06.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:06.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:06.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:06.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:06.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:06.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:06.291+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:06.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:06.291+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:06.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:06.291+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:06.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:06.291+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:06.505 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:06.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:06.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:06.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:06.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:06.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:06.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:06.504+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:07.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:07.011+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:07.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:07.011+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:07.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:07.011+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:07.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:07.011+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:07.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:07.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:07.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:07.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:07.254 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:07.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:07.254 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:07.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:07.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:07.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:07.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:07.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:07.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:07.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:07.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:07.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:07.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:07.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:07.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:07.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:07.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:07.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:07.538 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:07.537+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:08.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:08.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:08.247+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:08.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:08.247+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:08.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:08.247+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:08.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:08.247+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:08.280 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:08.279+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:08.280 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:08.279+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:08.280 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:08.279+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:08.281 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:08.279+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:08.521 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:08.521+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:08.522 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:08.521+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:08.522 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:08.521+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:08.522 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:08.521+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:09.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:09.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:09.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:09.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:09.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:09.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:09.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:09.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:09.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:09.233+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:09.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:09.233+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:09.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:09.233+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:09.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:09.233+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:09.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:09.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:09.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:09.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:09.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:09.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:09.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:09.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:09.530 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:09.529+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:09.530 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:09.529+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:09.530 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:09.529+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:09.530 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:09.529+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:10.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:10.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:10.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:10.018 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:10.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:10.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:10.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:10.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:10.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:10.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:10.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:10.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:10.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:10.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:10.317 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:10.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:10.317 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:10.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:10.317 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:10.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:10.530 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:10.530+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:10.531 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:10.530+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:10.531 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:10.530+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:10.531 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:10.530+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:10.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:10.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:10.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:10.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:10.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:11.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:11.272+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:11.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:11.272+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:11.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:11.272+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:11.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:11.272+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:11.321 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:11.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:11.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:11.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:11.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:11.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:11.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:11.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:11.512 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:11.512+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:11.512 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:11.512+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:11.513 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:11.512+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:11.513 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:11.512+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:11.955 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:11.955 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:11.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:11.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:12.255 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:12.254+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:12.255 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:12.254+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:12.255 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:12.254+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:12.255 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:12.254+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:12.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:12.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:12.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:12.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:12.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:12.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:12.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:12.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:12.506 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:12.506+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:12.507 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:12.506+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:12.507 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:12.506+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:12.507 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:12.506+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:12.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:12.960+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:12.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:12.960+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:12.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:12.960+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:12.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:12.960+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:13.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:13.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:13.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:13.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:13.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:13.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:13.221 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:13.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:13.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:13.341+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:13.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:13.341+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:13.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:13.341+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:13.343 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:13.341+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:13.490 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:13.489+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:13.490 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:13.489+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:13.490 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:13.489+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:13.490 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:13.489+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:13.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:13.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:13.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:13.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:13.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:13.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:13.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:13.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:14.245 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:14.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:14.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:14.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:14.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:14.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:14.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:14.245+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:14.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:14.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:14.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:14.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:14.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:14.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:14.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:14.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:14.531 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:14.531+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:14.531 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:14.531+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:14.532 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:14.531+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:14.532 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:14.531+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:14.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:14.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:14.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:14.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:14.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:14.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:14.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:14.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:15.237 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:15.236+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:15.237 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:15.236+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:15.237 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:15.236+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:15.237 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:15.236+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:15.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:15.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:15.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:15.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:15.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:15.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:15.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:15.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:15.500 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:15.499+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:15.500 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:15.499+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:15.500 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:15.499+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:15.500 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:15.499+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:15.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:15.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:15.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:15.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:15.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:15.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:15.929 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:15.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:16.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:16.220+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:16.221 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:16.220+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:16.221 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:16.220+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:16.221 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:16.220+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:16.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:16.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:16.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:16.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:16.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:16.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:16.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:16.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:16.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:16.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:16.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:16.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:16.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:16.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:16.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:16.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:16.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:16.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:16.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:16.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:16.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:16.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:16.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:16.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:17.194 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:17.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:17.195 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:17.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:17.195 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:17.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:17.195 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:17.194+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:17.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:17.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:17.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:17.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:17.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:17.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:17.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:17.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:17.438 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:17.438+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:17.438 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:17.438+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:17.439 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:17.438+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:17.439 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:17.438+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:17.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:17.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:17.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:17.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:17.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:17.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:17.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:17.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:18.230 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:18.229+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:18.230 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:18.229+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:18.230 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:18.229+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:18.230 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:18.229+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:18.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:18.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:18.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:18.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:18.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:18.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:18.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:18.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:18.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:18.483+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:18.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:18.483+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:18.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:18.483+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:18.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:18.483+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:18.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:18.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:18.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:18.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:18.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:18.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:18.997 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:18.996+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:19.188 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:19.187+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:19.188 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:19.187+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:19.188 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:19.187+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:19.188 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:19.187+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:19.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:19.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:19.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:19.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:19.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:19.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:19.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:19.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:19.497 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:19.497+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:19.498 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:19.497+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:19.498 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:19.497+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:19.498 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:19.497+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:19.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:19.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:19.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:19.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:19.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:19.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:19.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:19.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:20.195 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:20.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:20.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:20.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:20.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:20.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:20.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:20.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:20.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:20.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:20.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:20.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:20.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:20.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:20.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:20.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:20.519 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:20.518+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:20.519 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:20.518+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:20.519 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:20.518+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:20.519 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:20.518+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:20.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:20.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:20.947 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:20.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:20.947 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:20.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:20.947 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:20.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:21.156 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:21.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:21.157 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:21.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:21.157 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:21.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:21.157 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:21.155+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:21.346 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:21.344+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:21.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:21.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:21.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:21.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:21.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:21.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:21.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:21.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:21.484 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:21.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:21.485 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:21.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:21.485 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:21.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:21.485 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:21.484+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:21.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:21.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:21.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:21.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:21.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:21.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:21.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:21.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:22.161 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:22.160+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:22.161 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:22.160+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:22.161 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:22.160+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:22.162 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:22.160+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:22.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:22.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:22.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:22.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:22.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:22.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:22.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:22.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:22.482 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:22.481+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:22.483 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:22.481+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:22.483 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:22.481+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:22.483 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:22.481+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:22.860 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:22.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:22.860 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:22.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:22.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:22.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:22.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:22.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:23.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:23.131+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:23.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:23.131+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:23.131 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:23.131+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:23.132 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:23.131+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:23.415 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:23.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:23.415 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:23.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:23.415 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:23.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:23.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:23.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:23.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:23.474+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:23.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:23.474+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:23.475 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:23.474+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:23.475 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:23.474+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:23.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:23.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:23.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:23.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:23.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:23.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:23.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:23.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:24.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:24.083+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:24.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:24.083+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:24.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:24.083+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:24.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:24.083+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:24.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:24.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:24.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:24.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:24.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:24.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:24.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:24.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:24.444 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:24.444+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:24.444 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:24.444+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:24.445 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:24.444+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:24.445 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:24.444+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:24.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:24.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:24.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:24.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:24.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:24.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:24.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:24.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:25.089 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:25.088+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:25.089 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:25.088+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:25.089 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:25.088+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:25.089 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:25.088+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:25.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:25.396+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:25.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:25.396+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:25.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:25.396+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:25.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:25.396+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:25.420 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:25.420+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:25.420 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:25.420+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:25.420 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:25.420+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:25.421 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:25.420+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:25.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:25.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:25.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:25.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:25.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:25.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:25.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:25.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:26.056 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:26.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:26.056 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:26.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:26.056 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:26.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:26.057 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:26.056+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:26.346 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:26.345+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 9 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:26.376 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:26.376+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:26.376 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:26.376+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:26.376 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:26.376+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:26.376 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:26.376+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:26.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:26.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:26.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:26.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:26.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:26.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:26.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:26.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:26.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:26.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:26.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:26.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:26.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:26.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:26.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:26.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:27.093 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:27.093+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:27.093 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:27.093+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:27.094 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:27.093+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:27.094 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:27.093+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:27.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:27.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:27.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:27.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:27.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:27.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:27.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:27.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:27.437 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:27.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:27.437 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:27.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:27.437 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:27.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:27.437 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:27.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:27.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:27.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:27.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:27.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:27.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:27.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:27.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:27.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:28.085 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:28.085+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:28.086 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:28.085+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:28.086 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:28.085+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:28.086 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:28.085+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:28.353 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:28.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:28.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:28.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:28.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:28.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:28.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:28.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:28.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:28.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:28.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:28.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:28.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:28.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:28.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:28.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:28.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:28.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:28.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:28.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:28.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:28.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:28.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:28.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:29.042 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:29.041+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:29.042 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:29.041+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:29.043 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:29.041+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:29.043 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:29.041+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:29.364 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:29.362+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:29.364 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:29.362+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:29.364 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:29.362+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:29.364 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:29.362+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:29.441 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:29.440+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:29.441 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:29.440+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:29.441 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:29.440+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:29.442 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:29.440+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:29.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:29.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:29.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:29.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:29.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:29.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:29.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:29.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:30.026 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:30.024+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:30.026 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:30.024+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:30.026 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:30.024+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:30.026 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:30.024+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:30.401 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:30.400+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:30.402 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:30.400+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:30.402 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:30.400+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:30.402 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:30.400+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:30.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:30.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:30.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:30.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:30.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:30.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:30.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:30.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:30.883 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:30.882+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:30.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:30.882+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:30.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:30.882+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:30.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:30.882+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:31.035 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.034+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:31.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.034+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:31.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.034+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:31.036 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.034+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:31.346 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:31.345+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 20 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:31.433 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:31.431+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:31.433 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:31.431+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:31.433 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:31.431+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:31.433 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:31.431+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:31.505 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:31.504+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:31.505 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:31.504+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:31.505 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:31.504+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:31.506 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:31.504+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:31.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:31.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:31.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:31.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:31.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:31.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:31.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:31.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:31.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:31.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:31.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:31.997 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:31.995+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:32.386 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:32.385+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:32.387 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:32.385+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:32.387 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:32.385+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:32.387 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:32.385+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:32.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:32.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:32.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:32.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:32.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:32.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:32.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:32.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:32.833 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:32.832+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:32.834 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:32.832+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:32.834 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:32.832+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:32.834 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:32.832+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:33.002 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.001+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:33.002 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.001+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:33.002 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.001+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:33.002 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.001+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:33.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:33.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:33.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:33.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:33.354 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:33.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:33.355 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:33.353+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:33.465 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:33.464+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:33.466 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:33.464+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:33.466 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:33.464+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:33.466 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:33.464+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:33.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:33.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:33.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:33.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:33.854 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:33.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:33.854 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:33.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:33.958 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.957+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:33.959 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.957+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:33.959 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.957+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:33.959 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:33.957+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:34.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:34.318+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:34.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:34.318+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:34.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:34.318+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:34.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:34.318+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:34.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:34.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:34.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:34.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:34.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:34.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:34.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:34.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:34.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:34.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:34.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:34.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:34.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:34.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:34.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:34.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:34.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:34.993+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:34.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:34.993+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:34.994 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:34.993+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:34.995 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:34.993+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:35.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:35.319+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:35.320 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:35.319+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:35.321 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:35.319+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:35.321 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:35.319+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:35.489 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:35.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:35.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:35.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:35.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:35.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:35.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:35.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:35.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:35.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:35.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:35.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:35.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:35.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:35.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:35.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:36.007 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:36.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:36.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:36.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.006+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:36.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:36.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:36.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:36.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:36.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:36.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:36.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:36.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:36.346 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:36.345+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 28 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:36.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:36.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:36.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:36.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:36.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:36.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:36.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:36.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:36.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:36.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:36.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:36.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:36.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:36.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:36.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:36.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:37.000 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.999+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:37.000 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.999+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:37.000 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.999+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:37.000 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:36.999+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:37.264 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:37.263+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:37.264 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:37.263+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:37.265 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:37.263+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:37.265 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:37.263+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:37.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:37.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:37.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:37.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:37.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:37.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:37.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:37.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:37.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:37.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:37.918 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:37.917+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:38.016 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:38.015+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:38.016 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:38.015+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:38.017 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:38.015+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:38.017 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:38.015+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:38.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:38.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:38.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:38.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:38.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:38.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:38.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:38.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:38.503 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:38.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:38.503 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:38.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:38.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:38.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:38.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:38.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:38.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:38.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:38.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:38.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:38.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:38.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:38.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:38.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:39.054 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:39.053+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:39.054 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:39.053+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:39.055 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:39.053+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:39.055 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:39.053+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:39.246 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:39.245+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:39.246 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:39.245+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:39.246 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:39.245+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:39.246 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:39.245+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:39.461+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:39.461+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:39.461+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:39.461+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:39.836 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:39.835+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:39.836 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:39.835+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:39.837 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:39.835+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:39.837 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:39.835+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:40.022 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:40.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:40.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:40.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:40.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:40.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:40.023 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:40.021+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:40.203 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:40.202+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:40.203 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:40.202+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:40.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:40.202+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:40.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:40.202+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:40.498 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:40.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:40.498 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:40.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:40.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:40.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:40.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:40.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:40.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:40.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:40.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:40.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:40.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:40.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:40.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:40.884+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:41.043 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:41.042+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:41.044 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:41.042+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:41.044 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:41.042+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:41.044 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:41.042+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:41.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:41.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:41.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:41.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:41.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:41.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:41.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:41.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:41.347 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:41.345+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 50 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:41.526 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:41.525+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:41.526 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:41.525+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:41.526 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:41.525+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:41.526 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:41.525+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:41.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:41.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:41.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:41.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:41.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:41.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:41.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:41.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:42.020 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:42.019+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:42.020 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:42.019+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:42.021 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:42.019+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:42.021 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:42.019+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:42.138 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:42.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:42.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:42.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:42.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:42.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:42.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:42.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:42.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:42.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:42.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:42.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:42.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:42.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:42.499 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:42.497+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:42.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:42.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:42.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:42.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:42.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:42.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:42.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:42.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:43.021 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.020+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:43.021 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.020+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:43.021 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.020+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:43.022 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.020+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:43.099 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:43.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:43.100 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:43.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:43.100 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:43.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:43.100 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:43.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:43.478 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:43.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:43.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:43.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:43.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:43.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:43.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:43.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:43.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:43.995 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.994+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:43.995 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.994+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:36:40.282408+0000) 2023-07-04T23:38:43.995 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.994+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:43.995 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:43.994+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:44.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:44.093+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:44.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:44.093+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:44.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:44.093+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:44.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:44.093+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:44.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:44.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:44.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:44.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:44.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:44.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:44.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:44.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:44.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:44.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:44.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:44.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:44.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:44.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:36:40.932411+0000) 2023-07-04T23:38:44.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:44.908+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:45.011 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:45.010+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:45.012 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:45.010+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:45.066 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:45.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:45.066 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:45.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:45.066 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:45.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:45.066 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:45.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:45.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:45.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:45.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:45.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:45.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:45.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:45.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:45.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:45.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:45.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:45.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:45.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:45.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:45.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:45.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:45.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:46.085 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:46.084+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:46.085 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:46.084+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:46.085 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:46.084+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:46.085 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:46.084+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:46.347 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:46.346+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 69 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:46.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:46.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:46.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:46.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:46.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:46.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:46.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:46.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:46.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:46.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:46.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:46.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:46.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:46.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:46.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:46.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:47.066 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:47.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:47.067 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:47.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:47.067 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:47.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:47.067 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:47.065+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:47.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:47.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:47.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:47.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:47.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:47.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:47.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:47.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:47.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:47.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:47.850 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:47.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:48.001 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:48.000+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:48.002 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:48.000+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:48.088 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:48.087+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:48.088 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:48.087+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:48.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:48.087+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:48.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:48.087+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:48.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:48.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:48.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:48.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:48.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:48.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:48.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:48.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:48.863 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:48.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:48.863 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:48.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:49.051 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:49.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:49.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:49.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:49.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:49.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:49.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:49.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:49.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:49.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:49.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:49.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:49.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:49.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:49.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:49.826 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:49.826+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:49.827 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:49.826+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:50.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.074 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:50.073+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:50.075 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:50.073+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:50.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:50.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:50.388 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:50.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:50.388 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:50.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:50.388 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:50.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:50.806 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:50.805+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:50.806 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:50.805+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:50.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.966+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.966+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.966+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:50.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:50.966+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:51.076 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:51.076+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:51.077 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:51.076+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:51.348 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:51.347+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 99 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:51.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:51.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:51.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:51.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:51.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:51.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:51.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:51.373+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:51.834 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:51.833+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:51.834 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:51.833+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:51.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:51.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:51.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:51.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:51.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:51.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:51.965 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:51.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:52.052 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:52.052+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:52.053 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:52.052+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:52.363 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:52.362+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:52.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:52.362+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:52.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:52.362+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:36:35.626596+0000) 2023-07-04T23:38:52.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:52.362+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:52.787 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:52.786+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:52.788 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:38:52.786+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:36:36.231912+0000) 2023-07-04T23:38:52.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:52.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:52.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:52.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:52.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:52.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:52.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:52.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:53.049 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:53.048+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:53.049 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:38:53.048+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:36:36.181872+0000) 2023-07-04T23:38:53.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:53.408+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:53.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:53.897+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:54.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:54.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:54.865 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:54.864+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:55.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:55.372+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:55.870 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:55.869+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:56.348 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:38:56.347+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 123 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:38:56.415 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:56.414+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:56.870 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:56.869+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:57.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:57.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:57.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:57.917+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:58.406 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:58.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:58.886 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:58.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:38:59.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:38:59.438+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:38:59.840 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:38:59.839+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:00.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:00.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:00.823 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:00.823+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:01.349 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:01.347+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 148 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:01.394 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:01.393+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:01.822 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:01.821+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:02.379 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:02.378+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:02.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:02.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:03.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:03.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:03.817 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:03.816+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:04.401 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:04.400+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:04.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:04.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:05.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:05.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:36:46.128407+0000) 2023-07-04T23:39:05.633 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:05.632+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:05.633 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:05.632+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:05.815 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:05.815+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:06.349 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:06.348+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 174 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:06.665 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:06.665+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:06.666 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:06.665+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:06.823 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:06.822+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:06.985 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:06.984+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:06.985 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:06.984+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:07.632 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:07.632+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:07.633 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:07.632+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:07.787 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:07.786+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:07.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:07.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:07.971 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:07.970+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:08.590 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:08.589+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:08.590 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:08.589+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:08.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:08.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:08.966 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:08.966+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:08.966 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:08.966+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:09.630 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:09.630+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:09.631 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:09.630+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:09.833 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:09.832+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:09.979 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:09.978+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:09.979 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:09.978+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:10.645 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:10.645+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:10.645 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:10.645+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:10.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:10.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:10.940 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:10.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:10.940 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:10.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:11.349 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:11.349+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 199 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:11.609 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:11.608+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:11.609 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:11.608+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:11.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:11.855+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:11.922 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:11.922+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:11.922 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:11.922+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:12.654 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:12.654+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:12.654 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:12.654+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:12.807 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:12.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:12.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:12.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:13.673 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:13.672+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:13.673 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:13.672+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:13.795 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:13.795+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:13.903 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:13.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:13.903 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:13.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:14.490+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:14.490+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:14.490+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:14.679 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:14.679+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:14.679 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:14.679+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:14.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:14.756+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:14.952 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:14.951+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:14.952 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:14.952+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:14.952 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:14.952+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:14.952 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:14.952+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:15.470 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:15.470+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:15.470 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:15.470+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:15.470 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:15.470+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:15.471 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:15.470+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:15.720 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:15.720+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:15.720 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:15.720+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:15.721 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:15.720+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:15.721 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:15.720+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:15.754 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:15.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:15.947 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:15.946+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:15.947 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:15.946+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:15.947 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:15.946+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:15.947 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:15.946+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:16.350 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:16.349+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 223 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:16.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:16.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:16.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:16.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:16.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:16.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:16.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:16.492+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:16.720 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:16.719+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:16.721 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:16.719+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:16.721 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:16.719+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:16.721 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:16.719+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:16.728 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:16.726+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:16.929 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:16.928+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:16.929 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:16.928+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:16.929 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:16.928+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:16.929 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:16.928+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:17.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:17.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:17.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:17.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:17.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:17.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:17.498 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:17.496+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:17.740 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:17.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:17.740 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:17.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:17.740 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:17.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:17.740 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:17.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:17.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:17.754+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:17.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:17.754+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:17.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:17.754+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:17.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:17.754+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:17.904 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:17.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:17.905 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:17.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:17.905 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:17.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:17.905 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:17.903+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:18.459 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:18.458+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:18.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:18.458+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:18.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:18.458+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:18.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:18.458+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:18.723 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:18.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:18.724 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:18.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:18.724 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:18.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:18.724 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:18.722+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:18.775 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:18.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:18.775 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:18.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:18.775 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:18.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:18.775 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:18.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:18.862 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:18.860+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:18.862 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:18.860+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:18.862 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:18.860+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:18.862 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:18.860+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:19.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:19.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:19.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:19.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:19.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:19.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:19.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:19.460+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:19.742 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:19.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:19.743 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:19.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:19.743 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:19.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:19.743 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:19.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:19.795 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:19.794+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:19.796 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:19.794+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:19.796 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:19.794+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:19.796 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:19.794+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:19.817 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:19.816+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:19.817 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:19.816+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:19.817 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:19.816+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:19.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:19.816+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:20.492 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:20.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:20.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:20.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:20.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:20.491+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:20.493 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:20.491+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:20.775 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:20.774+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:20.775 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:20.774+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:20.775 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:20.774+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:20.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:20.774+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:20.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:20.789+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:20.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:20.789+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:20.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:20.789+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:20.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:20.789+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:20.811 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:20.810+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:20.811 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:20.810+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:20.811 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:20.810+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:20.811 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:20.810+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:21.350 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:21.349+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 250 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:21.513 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:21.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:21.513 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:21.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:21.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:21.512+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:21.514 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:21.512+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:21.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:21.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:21.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:21.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:21.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:21.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:21.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:21.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:21.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:21.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:21.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:21.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:21.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:21.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:21.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:21.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:21.766 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:21.765+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:21.766 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:21.765+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:21.766 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:21.765+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:21.767 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:21.765+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:22.510 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:22.508+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:22.510 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:22.508+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:22.510 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:22.508+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:22.510 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:22.508+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:22.731 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:22.730+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:22.731 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:22.730+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:22.731 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:22.730+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:22.732 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:22.730+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:22.797 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:22.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:22.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:22.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:22.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:22.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:22.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:22.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:22.799 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:22.798+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:22.800 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:22.798+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:22.800 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:22.798+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:22.800 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:22.798+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:23.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:23.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:23.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:23.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:23.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:23.502+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:23.504 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:23.502+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:23.699 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:23.698+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:23.699 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:23.698+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:23.699 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:23.698+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:23.699 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:23.698+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:23.829 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:23.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:23.829 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:23.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:23.829 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:23.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:23.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:23.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:23.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:23.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:23.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:23.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:23.839 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:23.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:23.840 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:23.838+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:24.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:24.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:24.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:24.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:24.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:24.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:24.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:24.534+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:24.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:24.744+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:24.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:24.744+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:24.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:24.744+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:24.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:24.744+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:24.815 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:24.814+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:24.815 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:24.814+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:24.815 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:24.814+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:24.815 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:24.814+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:24.841 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:24.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:24.841 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:24.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:24.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:24.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:24.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:24.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:25.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:25.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:25.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:25.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:25.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:25.534+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:25.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:25.534+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:25.703 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:25.702+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:25.703 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:25.702+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:25.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:25.702+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:25.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:25.702+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:25.821 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:25.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:25.822 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:25.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:25.822 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:25.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:25.822 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:25.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:25.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:25.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:25.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:25.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:25.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:25.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:25.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:25.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:26.351 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:26.349+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 276 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:26.571 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:26.570+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:26.571 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:26.570+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:26.571 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:26.570+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:26.571 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:26.570+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:26.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:26.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:26.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:26.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:26.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:26.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:26.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:26.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:26.793 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:26.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:26.793 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:26.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:26.793 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:26.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:26.794 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:26.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:26.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:26.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:26.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:26.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:26.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:26.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:26.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:26.867+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:27.524 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:27.523+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:27.524 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:27.523+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:27.524 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:27.523+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:27.524 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:27.523+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:27.525 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:27.523+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:27.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:27.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:27.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:27.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:27.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:27.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:27.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:27.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:27.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:27.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:27.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:27.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:27.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:27.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:27.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:27.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:27.863 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:27.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:27.863 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:27.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:27.863 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:27.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:27.864 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:27.862+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:28.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:28.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:28.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:28.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:28.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:28.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:28.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:28.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:28.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:28.474+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:28.635 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:28.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:28.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:28.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:28.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:28.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:28.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:28.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:28.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:28.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:28.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:28.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:28.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:28.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:28.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:28.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:28.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:28.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:28.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:28.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:28.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:28.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:28.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:28.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:29.467 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:29.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:29.467 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:29.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:29.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:29.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:29.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:29.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:29.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:29.466+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:29.671 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:29.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:29.671 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:29.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:29.671 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:29.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:29.671 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:29.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:29.794 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:29.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:29.794 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:29.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:29.794 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:29.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:29.794 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:29.792+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:29.852 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:29.851+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:29.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:29.851+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:29.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:29.851+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:29.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:29.851+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:30.461 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:30.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:30.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:30.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:30.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:30.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:30.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:30.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:30.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:30.460+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:30.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:30.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:30.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:30.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:30.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:30.834+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:30.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:30.834+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:30.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:30.834+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:30.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:30.834+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:30.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:30.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:30.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:30.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:30.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:30.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:30.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:30.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:31.351 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:31.350+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 301 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:31.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:31.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:31.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:31.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:31.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:31.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:31.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:31.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:31.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:31.442+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:31.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:31.738+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:31.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:31.738+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:31.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:31.738+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:31.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:31.738+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:31.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:31.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:31.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:31.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:31.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:31.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:31.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:31.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:31.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:31.935+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:31.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:31.935+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:31.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:31.935+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:31.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:31.935+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:32.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:32.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:32.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:32.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:32.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:32.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:32.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:32.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:32.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:32.395+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:32.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:32.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:32.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:32.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:32.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:32.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:32.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:32.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:32.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:32.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:32.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:32.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:32.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:32.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:32.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:32.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:32.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:32.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:32.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:32.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:32.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:32.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:32.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:32.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:33.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:33.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:33.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:33.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:33.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:33.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:33.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:33.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:33.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:33.380+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:33.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:33.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:33.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:33.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:33.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:33.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:33.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:33.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:33.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:33.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:33.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:33.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:33.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:33.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:33.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:33.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:33.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:33.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:33.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:33.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:33.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:33.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:33.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:33.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:34.354 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:34.353+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:34.355 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:34.353+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:34.355 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:34.353+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:34.355 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:34.353+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:34.355 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:34.353+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:34.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:34.724+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:34.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:34.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:34.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:34.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:34.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:34.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:34.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:34.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:34.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:34.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:34.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:34.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:34.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:34.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:34.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:34.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:34.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:34.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:34.982 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:34.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:34.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:34.981+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:35.326 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:35.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:35.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:35.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:35.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:35.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:35.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:35.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:35.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:35.325+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:35.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:35.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:35.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:35.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:35.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:35.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:35.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:35.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:35.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:35.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:35.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:35.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:35.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:35.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:35.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:35.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:35.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:35.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:35.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:35.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:35.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:35.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:35.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:35.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:36.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:36.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:36.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:36.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:36.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:36.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:36.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:36.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:36.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:36.308+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:36.352 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:36.351+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 325 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:36.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:36.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:36.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:36.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:36.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:36.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:36.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:36.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:36.940 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:36.939+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:36.940 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:36.939+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:36.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:36.939+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:36.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:36.939+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:36.959 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:36.958+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:36.959 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:36.958+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:36.959 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:36.958+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:36.959 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:36.958+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:37.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:37.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:37.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:37.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:37.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:37.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:37.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:37.308+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:37.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:37.308+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:37.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:37.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:37.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:37.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:37.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:37.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:37.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:37.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:37.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:37.940+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:37.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:37.940+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:37.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:37.940+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:37.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:37.940+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:37.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:37.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:37.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:37.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:37.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:37.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:37.986 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:37.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:38.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:38.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:38.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:38.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:38.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:38.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:38.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:38.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:38.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:38.346+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:38.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:38.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:38.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:38.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:38.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:38.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:38.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:38.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:38.923 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:38.922+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:38.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:38.922+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:38.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:38.922+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:38.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:38.922+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:38.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:38.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:38.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:38.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:38.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:38.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:38.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:38.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:39.394 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:39.393+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:39.394 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:39.393+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:39.394 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:39.393+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:39.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:39.393+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:39.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:39.393+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:39.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:39.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:39.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:39.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:39.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:39.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:39.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:39.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:39.962 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:39.961+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:39.962 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:39.961+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:39.963 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:39.961+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:39.963 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:39.961+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:39.983 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:39.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:39.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:39.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:39.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:39.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:39.984 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:39.982+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:40.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:40.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:40.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:40.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:40.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:40.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:40.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:40.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:40.436 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:40.434+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:40.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:40.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:40.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:40.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:40.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:40.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:40.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:40.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:40.916 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:40.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:40.916 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:40.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:40.916 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:40.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:40.916 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:40.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:40.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:41.352 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:41.351+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 349 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:41.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:41.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:41.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:41.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:41.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:41.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:41.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:41.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:41.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:41.417+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:41.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:41.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:41.766 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:41.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:41.766 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:41.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:41.766 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:41.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:41.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:41.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:41.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:41.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:41.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:41.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:41.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:41.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:41.942 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:41.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:41.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:41.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:41.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:41.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:41.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:41.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:42.459 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:42.459+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:42.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:42.459+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:42.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:42.459+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:42.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:42.459+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:42.460 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:42.459+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:42.812 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:42.811+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:42.812 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:42.811+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:42.812 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:42.811+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:42.812 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:42.811+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:42.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:42.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:42.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:42.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:42.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:42.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:42.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:42.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:42.921 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:42.920+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:42.921 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:42.920+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:42.921 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:42.920+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:42.921 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:42.920+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:43.476+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:43.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:43.476+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:43.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:43.842+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:43.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:43.842+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:43.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:43.842+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:43.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:43.842+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:43.873 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:43.872+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:43.873 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:43.872+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:43.873 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:43.872+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:43.873 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:43.872+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:43.874 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:43.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:43.874 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:43.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:43.875 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:43.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:43.875 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:43.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:44.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:44.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:44.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:44.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:44.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:44.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:44.451 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:44.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:44.451 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:44.449+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:44.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:44.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:44.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:44.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:44.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:44.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:44.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:44.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:44.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:44.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:44.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:44.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:44.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:44.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:44.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:44.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:44.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:44.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:44.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:44.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:44.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:44.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:44.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:44.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:45.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:45.410+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:45.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:45.410+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:45.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:45.410+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:45.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:45.410+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:45.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:45.410+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:45.807 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:45.806+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:45.807 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:45.806+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:45.807 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:45.806+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:45.807 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:45.806+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:45.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:45.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:45.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:45.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:45.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:45.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:45.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:45.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:45.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:45.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:45.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:45.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:45.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:45.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:45.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:45.869+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:46.353 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:46.352+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 376 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:46.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:46.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:46.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:46.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:46.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:46.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:46.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:46.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:46.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:46.366+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:46.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:46.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:46.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:46.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:46.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:46.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:46.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:46.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:46.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:46.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:46.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:46.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:46.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:46.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:46.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:46.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:46.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:46.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:46.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:46.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:46.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:46.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:46.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:46.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:47.358 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:47.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:47.358 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:47.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:47.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:47.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:47.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:47.358+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:47.359 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:47.358+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:47.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:47.743+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:47.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:47.743+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:47.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:47.743+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:47.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:47.743+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:47.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:47.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:47.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:47.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:47.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:47.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:47.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:47.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:47.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:47.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:47.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:47.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:47.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:47.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:47.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:47.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:48.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:48.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:48.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:48.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:48.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:48.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:48.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:48.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:48.319 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:48.317+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:48.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:48.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:48.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:48.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:48.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:48.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:48.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:48.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:48.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:48.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:48.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:48.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:48.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:48.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:48.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:48.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:48.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:48.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:48.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:48.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:48.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:48.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:48.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:48.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:49.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:49.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:49.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:49.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:49.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:49.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:49.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:49.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:49.312 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:49.309+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:49.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:49.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:49.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:49.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:49.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:49.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:49.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:49.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:49.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:49.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:49.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:49.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:49.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:49.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:49.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:49.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:36:35.949325+0000) 2023-07-04T23:39:49.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:49.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:49.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:49.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:49.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:49.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:49.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:49.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:50.347+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:50.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:50.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:50.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:50.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:50.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:50.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:50.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:50.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:50.812 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:50.811+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:50.812 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:50.811+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:50.812 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:50.811+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:50.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:50.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:50.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:50.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:50.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:50.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:50.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:50.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:51.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:51.345+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:51.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:51.345+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:51.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:51.345+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:51.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:51.345+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:51.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:51.345+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:51.354 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:51.353+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 400 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:51.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:51.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:51.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:51.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:51.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:51.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:51.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:51.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:51.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:51.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:51.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:51.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:51.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:51.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:51.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:51.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:51.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:51.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:51.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:51.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:51.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:51.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:52.314 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:52.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:52.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:52.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:52.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:52.313+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:52.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:52.746+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:52.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:52.746+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:52.747 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:52.746+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:52.747 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:52.746+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:52.814 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:52.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:52.814 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:52.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:52.814 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:52.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:52.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:52.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:52.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:52.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:52.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:52.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:52.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:52.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:53.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:53.327+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:53.328 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:53.327+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:53.328 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:53.327+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:53.328 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:53.327+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:53.328 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:53.327+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:53.790 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:53.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:53.790 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:53.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:53.791 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:53.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:53.791 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:53.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:53.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:53.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:53.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:53.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:53.814 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:53.813+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:53.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:53.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:53.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:53.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:53.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:53.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:53.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:53.928+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:54.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:54.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:54.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:54.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:54.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:54.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:54.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:54.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:54.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:54.370+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:54.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:54.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:54.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:54.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:54.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:54.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:54.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:54.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:54.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:54.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:54.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:54.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:54.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:54.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:54.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:54.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:54.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:54.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:54.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:54.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:54.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:54.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:55.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:55.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:55.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:55.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:55.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:55.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:55.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:55.403+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:55.404 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:55.403+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:55.825 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:55.824+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:55.825 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:55.824+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:55.825 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:55.824+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:55.825 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:55.824+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:55.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:55.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:55.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:55.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:55.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:55.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:55.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:55.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:55.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:55.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:55.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:55.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:55.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:55.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:56.354 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:39:56.353+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 425 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:39:56.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:56.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:56.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:56.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:56.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:56.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:56.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:56.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:56.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:56.424+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:56.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:56.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:56.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:56.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:56.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:56.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:56.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:56.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:56.876 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:56.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:56.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:56.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:56.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:56.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:56.953 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:56.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:56.953 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:56.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:56.954 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:56.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:56.954 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:56.953+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:57.390 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:57.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:57.390 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:57.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:57.391 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:57.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:57.391 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:57.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:57.391 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:57.390+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:57.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:57.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:57.774 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:57.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:57.774 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:57.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:57.774 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:57.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:57.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:57.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:57.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:57.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:57.864 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:57.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:57.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:57.927+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:57.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:57.927+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:57.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:57.927+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:57.928 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:57.927+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:58.365 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:58.364+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:58.365 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:58.364+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:58.365 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:58.364+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:58.365 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:58.364+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:58.365 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:58.364+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:58.740 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:58.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:58.740 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:58.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:58.740 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:58.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:58.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:58.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:58.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:58.825+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:58.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:58.825+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:58.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:58.825+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:58.901 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:58.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:58.901 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:58.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:58.901 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:58.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:58.901 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:58.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:59.317 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:59.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:59.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:59.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:59.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:59.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:39:59.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:59.317+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:39:59.318 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:39:59.317+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:39:59.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:59.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:59.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:59.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:39:59.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:59.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:59.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:39:59.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:39:59.838 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:59.838+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:59.838 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:59.838+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:59.839 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:39:59.838+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:39:59.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:59.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:59.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:59.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:39:59.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:59.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:39:59.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:39:59.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:00.297 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:00.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:00.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:00.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:00.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:00.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:00.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:00.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:00.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:00.297+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:00.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:00.726+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:00.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:00.726+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:00.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:00.726+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:00.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:00.726+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:00.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:00.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:00.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:00.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:00.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:00.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:00.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:00.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:00.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:00.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:00.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:00.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:00.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:00.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:01.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:01.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:01.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:01.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:01.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:01.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:01.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:01.309+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:01.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:01.309+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:01.355 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:01.355+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 449 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:01.695 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:01.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:01.695 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:01.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:01.695 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:01.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:01.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:01.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:01.821 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:01.821+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:01.821 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:01.821+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:01.821 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:01.821+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:01.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:01.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:01.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:01.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:01.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:01.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:01.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:01.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:02.340 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:02.340+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:02.341 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:02.340+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:02.341 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:02.340+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:02.341 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:02.340+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:02.341 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:02.340+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:02.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:02.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:02.686 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:02.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:02.686 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:02.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:02.686 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:02.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:02.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:02.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:02.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:02.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:02.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:02.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:02.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:02.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:02.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:02.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:02.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:02.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:02.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:02.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:03.294 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:03.294+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:03.294 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:03.294+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:03.295 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:03.294+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:03.295 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:03.294+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:03.295 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:03.294+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:03.660 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:03.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:03.660 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:03.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:03.660 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:03.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:03.660 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:03.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:03.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:03.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:03.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:03.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:03.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:03.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:03.942 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:03.942 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:03.942 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:03.941+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:03.942 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:03.941+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:04.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:04.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:04.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:04.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:04.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:04.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:04.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:04.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:04.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:04.263+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:04.652 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:04.651+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:04.653 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:04.651+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:04.653 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:04.651+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:04.654 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:04.651+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:04.847 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:04.845+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:04.847 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:04.845+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:04.848 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:04.845+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:04.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:04.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:04.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:04.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:04.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:04.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:04.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:04.906+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:05.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:05.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:05.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:05.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:05.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:05.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:05.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:05.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:05.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:05.259+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:05.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:05.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:05.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:05.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:05.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:05.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:05.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:05.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:05.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:05.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:05.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:05.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:05.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:05.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:05.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:05.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:05.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:05.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:05.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:05.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:05.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:05.926+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:06.296 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:06.295+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:06.296 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:06.295+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:06.296 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:06.295+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:06.297 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:06.295+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:06.297 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:06.295+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:06.355 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:06.354+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 476 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:06.656 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:06.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:06.657 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:06.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:06.657 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:06.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:06.657 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:06.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:06.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:06.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:06.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:06.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:06.880 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:06.880 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:06.881 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:06.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:07.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:07.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:07.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:07.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:07.279+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:07.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:07.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:07.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:07.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:07.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:07.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:07.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:07.634+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:07.884 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:07.883+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:07.884 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:07.883+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:07.884 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:07.883+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:07.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:07.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:07.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:07.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:07.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:07.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:07.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:07.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:08.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:08.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:08.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:08.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:08.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:08.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:08.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:08.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:08.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:08.278+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:08.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:08.637+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:08.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:08.637+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:08.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:08.637+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:08.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:08.637+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:08.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:08.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:08.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:08.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:08.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:08.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:08.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:08.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:08.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:08.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:08.926 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:08.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:08.927 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:08.925+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:09.204 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-07-04T23:40:09.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:09.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:09.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:09.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:09.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:09.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:09.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:09.315+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:09.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:09.315+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:09.323 DEBUG:tasks.ceph:Missed logrotate, host unreachable 2023-07-04T23:40:09.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:09.682+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:09.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:09.682+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:09.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:09.682+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:09.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:09.682+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:09.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-07-04T23:40:09.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-07-04T23:40:09.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-07-04T23:40:09.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-07-04T23:40:09.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-07-04T23:40:09.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-07-04T23:40:09.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-07-04T23:40:09.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-07-04T23:40:09.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-07-04T23:40:09.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:09.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:09.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:09.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:09.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:09.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:09.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:09.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:09.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:09.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:09.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:09.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:09.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:09.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:10.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:10.270+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:10.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:10.270+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:10.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:10.270+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:10.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:10.270+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:10.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:10.270+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:10.707 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:10.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:10.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:10.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:10.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:10.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:10.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:10.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:10.844 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:10.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:10.844 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:10.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:10.845 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:10.843+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:10.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:10.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:10.976 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:10.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:10.976 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:10.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:10.976 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:10.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:11.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:11.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:11.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:11.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:11.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:11.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:11.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:11.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:11.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:11.230+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:11.356 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:11.355+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 500 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:11.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:11.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:11.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:11.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:11.748 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:11.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:11.749 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:11.747+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:11.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:11.803+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:11.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:11.803+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:11.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:11.803+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:11.955 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:11.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:11.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:11.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:11.954+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:12.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:12.197+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:12.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:12.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:12.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:12.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:12.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:12.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:12.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:12.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:12.820 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:12.819+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:12.820 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:12.819+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:12.821 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:12.819+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:12.821 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:12.819+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:12.994 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:12.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:12.994 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:12.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:12.994 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:12.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:12.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:12.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:13.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:13.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:13.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:13.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:13.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:13.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:13.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:13.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:13.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:13.219+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:13.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:13.703+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:13.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:13.703+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:13.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:13.703+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:13.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:13.703+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:13.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:13.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:13.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:13.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:13.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:13.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:13.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:13.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:13.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:13.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:13.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:13.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:13.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:13.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:13.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:13.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:14.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:14.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:14.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:14.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:14.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:14.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:14.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:14.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:14.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:14.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:14.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:14.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:14.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:14.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:14.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:14.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:14.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:14.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:14.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:14.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:14.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:14.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:14.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:14.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:14.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:14.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:14.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:14.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:14.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:14.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:15.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:15.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:15.258 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:15.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:15.258 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:15.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:15.258 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:15.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:15.258 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:15.256+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:15.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2023-07-04T23:40:15.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~6s 2023-07-04T23:40:15.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~6s 2023-07-04T23:40:15.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~6s 2023-07-04T23:40:15.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:15.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:15.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:15.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:15.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:15.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:15.713 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:15.711+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:15.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:15.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:15.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:15.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:15.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:15.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:15.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:15.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:15.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:15.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:15.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:15.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:15.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:15.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:15.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:15.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:16.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:16.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:16.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:16.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:16.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:16.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:16.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:16.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:16.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:16.219+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:16.357 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:16.355+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 525 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:16.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:16.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:16.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:16.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:16.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:16.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:16.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:16.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:16.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:16.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:16.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:16.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:16.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:16.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:16.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:16.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:16.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:16.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:16.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:16.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:16.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:16.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:16.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:16.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:17.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:17.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:17.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:17.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:17.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:17.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:17.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:17.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:17.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:17.195+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:17.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:17.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:17.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:17.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:17.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:17.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:17.802 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:17.801+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:17.819 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:17.818+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:17.819 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:17.818+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:17.819 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:17.818+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:17.820 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:17.818+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:17.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:17.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:17.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:17.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:17.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:17.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:17.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:17.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:18.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:18.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:18.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:18.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:18.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:18.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:18.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:18.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:18.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:18.218+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:18.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:18.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:18.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:18.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:18.774 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:18.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:18.774 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:18.772+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:18.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:18.855+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:18.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:18.855+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:18.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:18.855+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:18.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:18.855+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:18.936 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:18.936+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:18.936 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:18.936+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:18.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:18.936+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:18.937 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:18.936+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:19.176 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:19.175+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:19.176 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:19.175+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:19.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:19.175+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:19.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:19.175+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:19.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:19.175+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:19.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:19.753+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:19.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:19.753+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:19.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:19.753+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:19.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:19.753+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:19.857 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:19.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:19.857 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:19.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:19.857 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:19.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:19.857 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:19.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:19.908 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:19.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:19.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:19.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:19.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:19.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:19.909 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:19.907+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:20.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:20.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:20.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:20.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:20.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:20.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:20.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:20.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:20.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:20.203+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:20.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:20.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:20.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:20.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:20.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:20.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:20.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:20.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:20.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:20.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:20.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:20.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:20.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:20.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:20.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:20.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:20.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:20.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:20.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:20.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:20.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:20.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:20.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:20.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:21.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:21.177+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:21.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:21.177+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:21.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:21.177+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:21.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:21.177+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:21.180 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:21.177+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:21.357 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:21.356+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 549 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:21.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2023-07-04T23:40:21.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~12s 2023-07-04T23:40:21.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2023-07-04T23:40:21.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~12s 2023-07-04T23:40:21.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~12s 2023-07-04T23:40:21.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~12s 2023-07-04T23:40:21.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~12s 2023-07-04T23:40:21.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~12s 2023-07-04T23:40:21.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~12s 2023-07-04T23:40:21.753 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:21.751+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:21.753 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:21.751+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:21.753 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:21.751+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:21.753 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:21.751+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:21.847 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:21.846+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:21.847 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:21.846+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:21.848 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:21.846+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:21.848 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:21.846+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:21.914 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:21.913+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:21.914 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:21.913+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:21.914 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:21.913+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:21.914 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:21.913+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:22.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:22.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:22.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:22.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:22.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:22.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:22.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:22.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:22.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:22.222+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:22.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:22.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:22.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:22.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:22.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:22.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:22.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:22.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:22.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:22.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:22.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:22.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:22.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:22.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:22.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:22.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:22.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:22.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:22.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:22.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:22.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:22.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:22.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:22.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:23.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:23.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:23.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:23.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:23.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:23.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:23.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:23.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:23.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:23.238+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:23.721 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:23.720+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:23.721 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:23.720+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:23.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:23.720+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:23.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:23.720+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:23.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:23.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:23.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:23.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:23.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:23.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:23.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:23.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:23.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:23.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:23.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:23.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:24.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:24.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:24.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:24.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:24.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:24.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:24.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:24.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:24.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:24.207+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:24.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:24.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:24.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:24.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:24.755 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:24.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:24.756 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:24.754+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:24.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:24.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:24.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:24.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:24.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:24.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:24.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:24.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:24.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:24.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:24.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:24.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:24.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:24.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:24.936 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:24.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:25.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:25.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:25.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:25.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:25.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:25.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:25.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:25.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:25.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:25.233+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:25.803 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:25.802+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:25.803 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:25.802+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:25.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:25.802+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:25.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:25.802+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:25.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:25.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:25.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:25.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:25.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:25.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:25.949 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:25.947+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:25.968 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:25.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:25.968 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:25.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:25.968 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:25.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:25.969 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:25.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:26.250 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:26.249+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:26.251 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:26.249+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:26.251 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:26.249+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:26.251 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:26.249+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:26.251 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:26.249+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:26.358 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:26.356+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 576 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:26.800 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:26.799+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:26.800 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:26.799+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:26.800 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:26.799+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:26.801 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:26.799+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:26.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:26.924+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:26.925 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:26.924+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:26.925 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:26.924+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:26.925 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:26.924+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:26.981 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:26.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:26.981 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:26.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:26.981 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:26.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:26.981 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:26.980+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:27.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~18s 2023-07-04T23:40:27.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~18s 2023-07-04T23:40:27.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~18s 2023-07-04T23:40:27.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:27.283+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:27.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:27.283+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:27.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:27.283+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:27.285 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:27.283+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:27.285 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:27.283+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:27.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:27.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:27.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:27.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:27.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:27.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:27.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:27.764+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:27.948 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:27.947+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:27.948 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:27.947+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:27.949 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:27.947+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:27.949 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:27.947+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:28.006 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:28.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:28.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:28.007 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:28.308 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:28.307+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:28.308 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:28.307+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:28.308 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:28.307+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:28.308 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:28.307+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:28.309 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:28.307+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:28.784 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:28.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:28.784 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:28.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:28.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:28.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:28.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:28.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:28.918 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:28.917+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:28.918 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:28.917+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:28.918 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:28.917+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:28.918 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:28.917+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:28.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:28.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:28.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:28.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:28.987+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:29.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:29.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:29.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:29.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:29.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:29.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:29.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:29.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:29.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:29.344+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:29.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:29.749+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:29.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:29.749+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:29.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:29.749+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:29.751 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:29.749+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:29.963 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:29.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:29.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:29.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:29.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:29.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:29.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:29.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:29.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:29.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:29.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:29.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:29.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:29.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:29.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:29.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:30.314 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:30.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:30.314 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:30.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:30.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:30.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:30.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:30.313+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:30.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:30.313+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:30.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:30.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:30.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:30.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:30.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:30.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:30.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:30.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:31.011 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.010+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.011 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.010+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.011 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.010+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.012 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.010+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:31.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:31.024+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:31.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:31.024+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:31.025 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:31.024+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:31.025 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:31.024+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:31.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:31.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:31.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:31.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:31.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:31.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:31.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:31.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:31.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:31.349+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:31.358 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:31.357+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 600 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:31.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:31.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:31.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:31.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:31.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:31.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:31.738 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:31.737+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:31.972 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.971+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.972 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.971+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.972 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.971+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:31.972 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:31.971+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:32.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:32.035+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:32.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:32.035+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:32.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:32.035+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:32.036 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:32.035+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:32.388 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:32.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:32.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:32.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:32.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:32.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:32.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:32.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:32.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:32.388+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:32.737 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:32.736+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:32.737 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:32.736+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:32.737 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:32.736+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:32.737 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:32.736+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:32.938 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:32.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:32.938 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:32.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:32.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:32.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:32.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:32.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:33.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~23s 2023-07-04T23:40:33.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~23s 2023-07-04T23:40:33.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~23s 2023-07-04T23:40:33.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~23s 2023-07-04T23:40:33.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~23s 2023-07-04T23:40:33.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~23s 2023-07-04T23:40:33.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~23s 2023-07-04T23:40:33.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~23s 2023-07-04T23:40:33.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~23s 2023-07-04T23:40:33.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:33.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:33.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:33.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:33.050 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:33.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:33.050 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:33.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:33.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:33.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:33.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:33.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:33.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:33.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:33.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:33.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:33.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:33.391+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:33.780 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:33.779+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:33.780 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:33.779+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:33.780 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:33.779+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:33.780 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:33.779+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:33.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:33.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:33.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:33.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:33.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:33.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:33.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:33.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:34.087 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:34.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:34.087 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:34.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:34.087 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:34.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:34.087 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:34.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:34.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:34.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:34.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:34.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:34.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:34.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:34.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:34.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:34.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:34.416+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:34.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:34.808+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:34.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:34.808+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:34.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:34.808+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:34.809 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:34.808+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:34.876 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:34.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:34.876 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:34.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:34.876 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:34.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:34.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:34.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:35.071 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:35.070+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:35.071 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:35.070+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:35.071 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:35.070+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:35.071 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:35.070+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:35.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:35.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:35.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:35.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:35.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:35.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:35.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:35.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:35.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:35.371+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:35.778 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:35.777+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:35.778 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:35.777+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:35.778 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:35.777+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:35.779 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:35.777+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:35.835 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:35.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:35.835 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:35.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:35.835 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:35.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:35.835 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:35.835+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:36.104 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:36.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:36.105 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:36.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:36.105 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:36.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:36.105 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:36.104+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:36.336 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:36.336+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:36.337 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:36.336+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:36.337 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:36.336+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:36.337 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:36.336+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:36.337 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:36.336+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:36.359 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:36.358+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 625 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:36.768 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:36.767+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:36.768 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:36.767+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:36.768 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:36.767+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:36.768 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:36.767+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:36.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:36.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:36.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:36.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:36.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:36.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:36.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:36.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:37.104 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:37.103+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:37.104 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:37.103+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:37.104 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:37.103+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:37.104 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:37.103+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:37.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:37.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:37.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:37.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:37.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:37.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:37.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:37.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:37.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:37.366+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:37.770 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:37.769+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:37.770 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:37.769+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:37.770 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:37.769+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:37.770 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:37.769+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:37.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:37.831+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:37.832 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:37.831+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:37.832 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:37.831+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:37.832 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:37.831+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:38.146 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:38.146+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:38.147 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:38.146+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:38.147 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:38.146+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:38.147 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:38.146+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:38.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:38.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:38.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:38.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:38.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:38.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:38.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:38.392+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:38.394 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:38.392+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:38.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:38.761+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:38.762 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:38.761+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:38.762 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:38.761+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:38.762 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:38.761+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:38.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:38.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:38.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:38.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:38.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:38.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:38.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:38.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:38.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~29s 2023-07-04T23:40:38.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~29s 2023-07-04T23:40:38.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~29s 2023-07-04T23:40:38.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~29s 2023-07-04T23:40:38.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~29s 2023-07-04T23:40:38.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~29s 2023-07-04T23:40:38.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~29s 2023-07-04T23:40:38.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~29s 2023-07-04T23:40:38.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~29s 2023-07-04T23:40:39.195 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:39.194+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:39.195 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:39.194+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:39.195 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:39.194+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:39.195 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:39.194+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:39.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:40:39.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-07-04T23:40:39.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-04T23:40:39.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:39.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:39.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:39.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:39.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:39.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:39.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:39.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:39.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:39.380+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:39.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:39.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:39.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:39.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:39.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:39.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:39.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:39.719+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:39.725 DEBUG:teuthology.orchestra.run.smithi082:> true 2023-07-04T23:40:39.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:39.802+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:39.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:39.802+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:39.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:39.802+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:39.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:39.802+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:39.992 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2023-07-04T23:40:39.992 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:40:40.206 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:40.206+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:40.206 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:40.206+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:40.206 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:40.206+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:40.206 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:40.206+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:40.352 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:40.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:40.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:40.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:40.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:40.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:40.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:40.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:40.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:40.352+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:40.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:40.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:40.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:40.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:40.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:40.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:40.677 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:40.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:40.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:40.824+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:40.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:40.824+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:40.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:40.824+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:40.825 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:40.824+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:41.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:41.227+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:41.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:41.227+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:41.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:41.227+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:41.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:41.227+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:41.360 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:41.359+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 649 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:41.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:41.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:41.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:41.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:41.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:41.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:41.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:41.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:41.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:41.394+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:41.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:41.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:41.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:41.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:41.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:41.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:41.711 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:41.710+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:41.808 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:41.807+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:41.808 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:41.807+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:41.808 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:41.807+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:41.808 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:41.807+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:42.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:42.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:42.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:42.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:42.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:42.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:42.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:42.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:42.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:42.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:42.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:42.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:42.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:42.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:42.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:42.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:42.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:42.413+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:42.715 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:42.715+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:42.715 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:42.715+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:42.715 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:42.715+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:42.715 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:42.715+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:42.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:42.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:42.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:42.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:42.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:42.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:42.829 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:42.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:43.167 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:43.166+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:43.167 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:43.166+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:43.167 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:43.166+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:43.167 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:43.166+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:43.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:43.397+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:43.398 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:43.397+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:43.398 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:43.397+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:43.398 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:43.397+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:43.398 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:43.397+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:43.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:43.687+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:43.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:43.687+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:43.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:43.687+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:43.689 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:43.687+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:43.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:43.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:43.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:43.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:43.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:43.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:43.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:43.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:44.196 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:44.196+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:44.196 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:44.196+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:44.196 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:44.196+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:44.196 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:44.196+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:44.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:44.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:44.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:44.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:44.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:44.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:44.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:44.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:44.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:44.409+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:44.643 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:44.641+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:44.643 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:44.641+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:44.643 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:44.642+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:44.643 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:44.642+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:44.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~35s 2023-07-04T23:40:44.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~35s 2023-07-04T23:40:44.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~35s 2023-07-04T23:40:44.866 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:44.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:44.866 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:44.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:44.866 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:44.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:44.866 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:44.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:45.219 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:45.219+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:45.219 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:45.219+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:45.219 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:45.219+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:45.219 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:45.219+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:45.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:45.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:45.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:45.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:45.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:45.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:45.441 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:45.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:45.441 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:45.439+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:45.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:45.644+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:45.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:45.644+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:45.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:45.644+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:45.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:45.644+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:45.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:45.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:45.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:45.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:45.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:45.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:45.843 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:45.842+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:46.171 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:46.171+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:46.171 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:46.171+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:46.172 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:46.171+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:46.172 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:46.171+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:46.360 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:46.359+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 716 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:46.406 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:46.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:46.406 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:46.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:46.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:46.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:46.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:46.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:46.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:46.406+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:46.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:46.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:46.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:46.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:46.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:46.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:46.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:46.685+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:46.796 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:46.795+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:46.796 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:46.795+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:46.796 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:46.795+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:46.796 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:46.795+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:47.165 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:47.164+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:47.165 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:47.164+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:47.165 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:47.164+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:47.165 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:47.164+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:47.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:47.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:47.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:47.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:47.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:47.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:47.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:47.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:47.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:47.380+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:47.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:47.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:47.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:47.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:47.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:47.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:47.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:47.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:47.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:47.756+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:47.757 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:47.756+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:47.757 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:47.756+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:47.757 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:47.756+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:48.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:48.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:48.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:48.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:48.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:48.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:48.145 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:48.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:48.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:48.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:48.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:48.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:48.346 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:48.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:48.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:48.346+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:48.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:48.346+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:48.620 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:48.620+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:48.620 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:48.620+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:48.621 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:48.620+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:48.621 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:48.620+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:48.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:48.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:48.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:48.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:48.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:48.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:48.756 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:48.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:49.100 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:49.100+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:49.100 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:49.100+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:49.100 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:49.100+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:49.101 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:49.100+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:49.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:49.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:49.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:49.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:49.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:49.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:49.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:49.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:49.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:49.395+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:49.573 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:49.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:49.574 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:49.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:49.574 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:49.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:49.574 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:49.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:49.761 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:49.761+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:49.761 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:49.761+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:49.762 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:49.761+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:49.762 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:49.761+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:50.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:50.053+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:50.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:50.053+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:50.054 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:50.053+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:50.054 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:50.053+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:50.426 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:50.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:50.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:50.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:50.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:50.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:50.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:50.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:50.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:50.426+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:50.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~41s 2023-07-04T23:40:50.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~41s 2023-07-04T23:40:50.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~41s 2023-07-04T23:40:50.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~41s 2023-07-04T23:40:50.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~41s 2023-07-04T23:40:50.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~41s 2023-07-04T23:40:50.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~41s 2023-07-04T23:40:50.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~41s 2023-07-04T23:40:50.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~41s 2023-07-04T23:40:50.528 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:50.527+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:50.528 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:50.527+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:50.528 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:50.528+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:50.528 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:50.528+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:50.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:50.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:50.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:50.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:50.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:50.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:50.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:50.755+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:51.054 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:51.054+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:51.055 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:51.054+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:51.055 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:51.054+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:51.055 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:51.054+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:51.361 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:51.360+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 760 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:51.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:51.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:51.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:51.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:51.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:51.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:51.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:51.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:51.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:51.422+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:51.514+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:51.514+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:51.514+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:51.516 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:51.514+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:51.719 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:51.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:51.719 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:51.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:51.719 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:51.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:51.720 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:51.718+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:52.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:52.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:52.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:52.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:52.068 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:52.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:52.068 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:52.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:52.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:52.407+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:52.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:52.407+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:52.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:52.407+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:52.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:52.407+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:52.409 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:52.407+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:52.549 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:52.548+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:52.549 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:52.548+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:52.550 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:52.548+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:52.550 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:52.548+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:52.732 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:52.730+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:52.732 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:52.731+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:52.732 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:52.731+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:52.732 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:52.731+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:53.023 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:53.022+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:53.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:53.022+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:53.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:53.022+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:53.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:53.022+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:53.402 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:53.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:53.402 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:53.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:53.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:53.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:53.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:53.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:53.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:53.401+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:53.542 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:53.540+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:53.542 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:53.540+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:53.542 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:53.540+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:53.542 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:53.540+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:53.739 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:53.737+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:53.739 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:53.737+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:53.739 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:53.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:53.739 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:53.738+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:54.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:54.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:54.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:54.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:54.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:54.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:54.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:54.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:54.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:54.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:54.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:54.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:54.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:54.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:54.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:54.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:54.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:54.419+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:54.534 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:54.533+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:54.534 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:54.533+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:54.534 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:54.533+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:54.534 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:54.533+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:54.744 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:54.742+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:54.744 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:54.743+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:54.744 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:54.743+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:54.744 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:54.743+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:55.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:55.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:55.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:55.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:55.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:55.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:55.053 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:55.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:55.427 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:55.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:55.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:55.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:55.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:55.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:55.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:55.426+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:55.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:55.426+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:55.494 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:55.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:55.494 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:55.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:55.494 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:55.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:55.494 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:55.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:55.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:55.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:55.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:55.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:55.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:55.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:55.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:55.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:56.016 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.015+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:56.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.015+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:56.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.015+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:56.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.015+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:56.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~47s 2023-07-04T23:40:56.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~47s 2023-07-04T23:40:56.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~47s 2023-07-04T23:40:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~47s 2023-07-04T23:40:56.361 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:40:56.360+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 885 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:40:56.449 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:56.448+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:56.450 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:56.448+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:56.450 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:56.448+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:56.450 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:56.448+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:56.475 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:56.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:56.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:56.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:56.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:56.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:56.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:56.474+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:56.476 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:56.474+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:56.763 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:56.762+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:56.763 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:56.762+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:56.763 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:56.762+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:56.763 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:56.762+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:56.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:56.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:56.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:56.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:56.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:57.413 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:57.412+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:57.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:57.412+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:57.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:57.412+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:57.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:57.412+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:57.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:57.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:57.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:57.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:57.428 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:57.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:57.429 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:57.427+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:57.429 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:57.427+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:57.786 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:57.784+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:57.786 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:57.784+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:57.786 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:57.784+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:57.786 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:57.784+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:58.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:58.011 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:58.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:58.012 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.010+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:58.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:58.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:58.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:58.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:58.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:58.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:58.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:58.421+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:58.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:58.421+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:58.434 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:58.433+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:58.434 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:58.433+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:58.435 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:58.433+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:58.435 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:58.433+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:58.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:58.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:58.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:58.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:58.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:58.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:58.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:58.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:58.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:58.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:58.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:58.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:58.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:59.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:59.410+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:59.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:59.410+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:40:59.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:59.410+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:59.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:40:59.410+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:40:59.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:59.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:59.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:59.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:59.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:59.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:40:59.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:59.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:40:59.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:40:59.431+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:40:59.759 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:59.759+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:59.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:59.759+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:59.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:59.759+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:40:59.760 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:40:59.759+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:40:59.947 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:59.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:59.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:59.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:40:59.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:59.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:40:59.948 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:40:59.946+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:00.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:00.411+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:00.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:00.411+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:00.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:00.411+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:00.412 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:00.411+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:00.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:00.428+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:00.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:00.428+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:00.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:00.428+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:00.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:00.428+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:00.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:00.428+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:00.713 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:00.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:00.713 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:00.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:00.713 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:00.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:00.713 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:00.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:00.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:00.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:00.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:00.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:00.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:00.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:00.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:00.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:01.362 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:01.361+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 959 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:01.384 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:01.384 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:01.384 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:01.385 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:01.383+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:01.715 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:01.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:01.716 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:01.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:01.716 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:01.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:01.716 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:01.712+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:01.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:01.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:01.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:01.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:01.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:01.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:01.976 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:01.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:02.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~52s 2023-07-04T23:41:02.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~52s 2023-07-04T23:41:02.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~52s 2023-07-04T23:41:02.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~52s 2023-07-04T23:41:02.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~52s 2023-07-04T23:41:02.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~52s 2023-07-04T23:41:02.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~52s 2023-07-04T23:41:02.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~52s 2023-07-04T23:41:02.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~52s 2023-07-04T23:41:02.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:02.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:02.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:02.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:02.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:02.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:02.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:02.359+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:02.361 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:02.359+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:02.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:02.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:02.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:02.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:02.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:02.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:02.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:02.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:02.679 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:02.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:02.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:02.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:02.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:02.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:02.680 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:02.679+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:02.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:02.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:02.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:02.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:02.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:02.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:02.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:02.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:03.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:03.371+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:03.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:03.371+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:03.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:03.371+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:03.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:03.371+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:03.395 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:03.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:03.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:03.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:03.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:03.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:03.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:03.394+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:03.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:03.394+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:03.699 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:03.698+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:03.699 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:03.698+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:03.700 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:03.698+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:03.700 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:03.698+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:03.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:03.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:03.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:03.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:03.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:04.391 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:04.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:04.391 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:04.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:04.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:04.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:04.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:04.390+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:04.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:04.390+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:04.413 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:04.413+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:04.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:04.413+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:04.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:04.413+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:04.414 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:04.413+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:04.705 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:04.704+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:04.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:04.704+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:04.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:04.704+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:04.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:04.704+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:04.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:04.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:04.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:04.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:04.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:04.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:04.958 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:04.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:05.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:05.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:05.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:05.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:05.354 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:05.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:05.354 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:05.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:05.354 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:05.352+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:05.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:05.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:05.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:05.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:05.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:05.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:05.373 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:05.372+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:05.749 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:05.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:05.750 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:05.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:05.750 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:05.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:05.750 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:05.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:05.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:05.994+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:05.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:05.994+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:05.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:05.994+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:05.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:05.994+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:06.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:06.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:06.312 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:06.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:06.312 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:06.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:06.312 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:06.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:06.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:06.310+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:06.362 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:06.361+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1041 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:06.390 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:06.389+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:06.390 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:06.389+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:06.390 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:06.389+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:06.391 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:06.389+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:06.749 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:06.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:06.749 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:06.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:06.749 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:06.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:06.749 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:06.748+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:06.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:06.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:06.963 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:06.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:06.963 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:06.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:06.963 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:06.962+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:07.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:07.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:07.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:07.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:07.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:07.279+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:07.427 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:07.426+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:07.427 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:07.426+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:07.427 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:07.426+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:07.427 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:07.426+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:07.704 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:07.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:07.704 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:07.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:07.704 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:07.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:07.704 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:07.703+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:07.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~58s 2023-07-04T23:41:07.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~58s 2023-07-04T23:41:07.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~58s 2023-07-04T23:41:07.886 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~58s 2023-07-04T23:41:07.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:07.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:07.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:07.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:07.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:07.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:07.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:07.931+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:08.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:08.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:08.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:08.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:08.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:08.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:08.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:08.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:08.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:08.238+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:08.404 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:08.403+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:08.404 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:08.403+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:08.404 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:08.403+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:08.404 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:08.403+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:08.736 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:08.735+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:08.736 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:08.735+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:08.736 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:08.735+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:08.737 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:08.735+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:08.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:08.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:08.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:08.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:08.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:08.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:08.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:08.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:09.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:09.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:09.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:09.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:09.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:09.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:09.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:09.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:09.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:09.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:09.419 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:09.418+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:09.419 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:09.418+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:09.419 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:09.418+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:09.419 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:09.418+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:09.705 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:09.705+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:09.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:09.705+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:09.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:09.705+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:09.706 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:09.705+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:09.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:09.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:09.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:09.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:09.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:09.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:09.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:09.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:10.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:41:10.193 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:41:10.211 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:10.210+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:10.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:10.210+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:10.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:10.210+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:10.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:10.210+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:10.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:10.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:10.395 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:10.394+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:10.395 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:10.394+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:10.395 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:10.394+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:10.396 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:10.394+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:10.715 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:10.714+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:10.715 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:10.714+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:10.715 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:10.714+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:10.715 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:10.714+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:10.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:10.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:10.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:10.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:10.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:10.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:10.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:10.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:11.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:11.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:11.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:11.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:11.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:11.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:11.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:11.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:11.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:11.205+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:11.363 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:11.362+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1099 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:11.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:11.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:11.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:11.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:11.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:11.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:11.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:11.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:11.753 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:11.752+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:11.753 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:11.752+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:11.754 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:11.752+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:11.754 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:11.752+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:11.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:11.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:11.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:11.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:11.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:11.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:11.921 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:11.920+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:12.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:12.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:12.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:12.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:12.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:12.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:12.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:12.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:12.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:12.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:12.443 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:12.442+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:12.443 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:12.442+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:12.443 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:12.442+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:12.443 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:12.442+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:12.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:12.757+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:12.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:12.757+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:12.759 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:12.757+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:12.759 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:12.757+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:12.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:12.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:12.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:12.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:12.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:12.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:12.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:12.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:13.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:13.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:13.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:13.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:13.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:13.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:13.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:13.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:13.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:13.279+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:13.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:13.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:13.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:13.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:13.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:13.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:13.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:13.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:13.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~64s 2023-07-04T23:41:13.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~64s 2023-07-04T23:41:13.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~64s 2023-07-04T23:41:13.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~64s 2023-07-04T23:41:13.754 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:13.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:13.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:13.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:13.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:13.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:13.755 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:13.754+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:13.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:13.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:13.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:13.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:13.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:13.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:13.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:13.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:14.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:14.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:14.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:14.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:14.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:14.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:14.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:14.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:14.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:14.273+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:14.468 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:14.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:14.469 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:14.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:14.469 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:14.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:14.469 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:14.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:14.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:14.801+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:14.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:14.801+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:14.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:14.801+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:14.802 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:14.801+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:14.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:14.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:14.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:14.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:14.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:14.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:14.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:14.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:15.252 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:15.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:15.252 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:15.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:15.252 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:15.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:15.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:15.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:15.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:15.251+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:15.514 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:15.513+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:15.514 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:15.513+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:15.514 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:15.513+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:15.514 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:15.513+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:15.823 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:15.822+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:15.824 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:15.822+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:15.824 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:15.822+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:15.824 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:15.822+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:15.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:15.955+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:15.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:15.955+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:15.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:15.955+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:15.956 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:15.955+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:16.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:16.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:16.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:16.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:16.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:16.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:16.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:16.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:16.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:16.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:16.364 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:16.363+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1140 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:16.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:16.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:16.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:16.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:16.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:16.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:16.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:16.478+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:16.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:16.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:16.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:16.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:16.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:16.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:16.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:16.830+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:16.966 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:16.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:16.966 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:16.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:16.966 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:16.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:16.966 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:16.965+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:17.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:17.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:17.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:17.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:17.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:17.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:17.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:17.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:17.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:17.259+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:17.465 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:17.464+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:17.465 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:17.464+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:17.465 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:17.464+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:17.465 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:17.464+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:17.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:17.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:17.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:17.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:17.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:17.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:17.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:17.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:17.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:17.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:17.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:17.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:17.979 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:17.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:17.980 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:17.978+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:18.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:18.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:18.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:18.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:18.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:18.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:18.281 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:18.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:18.281 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:18.279+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:18.506 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:18.506+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:18.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:18.506+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:18.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:18.506+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:18.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:18.506+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:18.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:18.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:18.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:18.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:18.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:18.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:18.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:18.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:19.020 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:19.019+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:19.020 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:19.019+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:19.020 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:19.019+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:19.020 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:19.019+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:19.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:19.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:19.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:19.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:19.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:19.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:19.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:19.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:19.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:19.271+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:19.495 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:19.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:19.495 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:19.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:19.495 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:19.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:19.495 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:19.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~70s 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~70s 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~70s 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~70s 2023-07-04T23:41:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~70s 2023-07-04T23:41:19.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~70s 2023-07-04T23:41:19.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~70s 2023-07-04T23:41:19.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~70s 2023-07-04T23:41:19.903 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:19.902+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:19.903 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:19.902+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:19.903 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:19.902+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:19.903 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:19.902+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:20.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:20.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:20.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:20.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:20.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:20.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:20.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:20.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:20.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:20.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:20.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:20.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:20.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:20.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:20.502 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:20.501+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:20.502 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:20.501+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:20.502 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:20.501+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:20.502 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:20.501+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:20.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:20.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:20.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:20.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:20.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:20.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:20.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:20.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:20.964 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.963+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:20.965 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.963+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:20.965 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.963+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:20.965 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:20.963+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:21.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:21.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:21.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:21.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:21.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:21.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:21.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:21.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:21.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:21.222+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:21.365 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:21.363+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1194 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:21.537 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:21.536+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:21.537 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:21.536+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:21.537 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:21.536+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:21.538 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:21.536+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:21.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:21.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:21.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:21.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:21.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:21.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:21.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:21.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:21.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:21.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:21.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:21.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:22.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:22.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:22.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:22.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:22.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:22.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:22.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:22.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:22.266 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:22.264+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:22.492 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:22.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:22.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:22.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:22.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:22.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:22.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:22.492+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:22.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:22.905+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:22.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:22.905+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:22.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:22.905+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:22.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:22.905+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:22.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:22.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:22.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:22.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:22.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:22.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:22.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:22.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:23.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:23.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:23.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:23.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:23.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:23.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:23.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:23.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:23.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:23.236+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:23.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:23.462+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:23.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:23.462+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:23.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:23.462+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:23.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:23.462+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:23.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:23.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:23.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:23.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:23.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:23.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:23.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:23.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:23.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:24.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:24.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:24.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:24.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:24.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:24.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:24.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:24.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:24.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:24.213+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:24.488 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:24.487+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:24.488 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:24.487+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:24.488 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:24.487+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:24.488 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:24.487+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:24.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:24.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:24.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:24.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:24.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:24.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:24.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:24.891+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:24.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:24.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:24.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:24.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:24.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:24.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:24.907 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:24.905+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:25.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:25.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:25.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:25.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:25.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:25.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:25.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:25.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:25.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:25.188+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:25.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~76s 2023-07-04T23:41:25.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~76s 2023-07-04T23:41:25.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~76s 2023-07-04T23:41:25.471 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:25.470+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:25.471 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:25.470+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:25.471 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:25.470+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:25.471 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:25.470+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:25.882 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:25.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:25.882 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:25.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:25.882 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:25.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:25.882 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:25.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:25.901 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:25.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:25.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:25.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:25.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:25.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:25.902 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:25.901+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:26.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:26.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:26.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:26.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:26.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:26.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:26.194 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:26.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:26.194 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:26.192+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:26.365 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:26.364+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1241 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:26.431 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:26.431+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:26.432 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:26.431+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:26.432 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:26.431+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:26.432 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:26.431+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:26.892 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:26.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:26.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:26.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:26.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:26.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:26.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:26.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:26.911 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:26.911+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:26.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:26.911+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:26.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:26.911+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:26.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:26.911+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:27.161 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:27.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:27.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:27.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:27.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:27.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:27.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:27.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:27.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:27.161+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:27.434 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:27.434+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:27.434 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:27.434+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:27.435 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:27.434+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:27.435 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:27.434+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:27.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:27.885+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:27.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:27.885+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:27.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:27.885+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:27.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:27.885+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:27.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:27.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:27.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:27.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:27.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:27.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:27.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:27.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:28.182 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:28.181+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:28.182 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:28.181+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:28.182 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:28.181+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:28.182 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:28.181+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:28.182 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:28.181+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:28.478 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:28.477+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:28.478 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:28.477+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:28.478 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:28.477+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:28.478 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:28.477+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:28.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:28.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:28.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:28.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:28.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:28.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:28.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:28.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:28.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:28.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:28.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:28.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:28.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:28.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:28.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:28.932+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:29.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:29.226+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:29.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:29.226+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:29.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:29.226+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:29.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:29.226+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:29.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:29.226+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:29.443 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:29.443+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:29.444 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:29.443+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:29.444 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:29.443+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:29.444 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:29.443+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:29.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:29.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:29.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:29.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:29.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:29.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:29.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:29.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:29.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:29.914+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:29.915 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:29.914+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:29.915 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:29.914+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:29.915 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:29.914+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:30.190 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:30.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:30.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:30.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:30.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:30.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:30.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:30.190+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:30.191 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:30.190+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:30.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:30.466+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:30.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:30.466+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:30.467 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:30.466+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:30.467 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:30.466+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:30.864 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:30.863+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:30.864 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:30.863+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:30.864 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:30.863+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:30.864 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:30.863+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:30.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:30.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:30.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:30.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:30.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:30.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:30.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:30.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:31.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~81s 2023-07-04T23:41:31.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~81s 2023-07-04T23:41:31.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~81s 2023-07-04T23:41:31.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:31.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:31.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:31.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:31.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:31.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:31.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:31.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:31.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:31.225+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:31.366 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:31.365+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1280 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:31.516 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:31.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:31.516 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:31.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:31.516 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:31.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:31.516 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:31.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:31.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:31.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:31.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:31.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:31.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:31.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:31.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:31.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:31.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:31.950+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:31.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:31.950+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:31.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:31.950+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:31.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:31.950+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:32.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:32.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:32.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:32.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:32.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:32.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:32.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:32.188+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:32.189 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:32.188+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:32.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:32.559+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:32.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:32.559+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:32.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:32.559+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:32.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:32.559+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:32.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:32.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:32.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:32.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:32.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:32.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:32.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:32.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:32.954 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:32.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:32.955 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:32.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:32.955 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:32.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:32.955 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:32.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:33.152 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:33.152+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:33.152 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:33.152+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:33.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:33.152+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:33.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:33.152+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:33.153 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:33.152+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:33.565 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:33.565+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:33.566 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:33.565+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:33.566 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:33.565+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:33.566 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:33.565+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:33.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:33.870+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:33.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:33.870+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:33.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:33.870+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:33.870 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:33.870+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:33.973 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:33.969+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:33.973 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:33.969+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:33.973 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:33.969+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:33.973 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:33.969+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:34.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:34.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:34.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:34.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:34.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:34.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:34.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:34.200+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:34.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:34.200+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:34.545 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:34.544+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:34.546 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:34.544+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:34.546 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:34.544+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:34.546 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:34.544+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:34.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:34.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:34.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:34.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:34.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:34.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:34.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:34.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:34.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:34.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:34.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:34.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:34.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:34.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:34.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:34.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:35.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:35.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:35.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:35.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:35.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:35.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:35.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:35.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:35.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:35.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:35.509 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:35.509+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:35.510 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:35.509+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:35.510 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:35.509+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:35.510 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:35.509+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:35.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:35.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:35.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:35.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:35.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:35.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:35.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:35.874+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:35.955 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:35.955+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:35.956 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:35.955+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:35.956 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:35.955+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:35.956 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:35.955+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:36.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:36.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:36.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:36.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:36.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:36.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:36.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:36.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:36.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:36.245+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:36.367 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:36.367+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1332 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:36.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:36.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:36.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:36.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:36.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:36.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:36.521 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:36.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:36.855 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:36.854+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:36.855 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:36.854+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:36.855 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:36.854+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:36.855 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:36.854+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:36.909 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:36.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:36.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:36.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:36.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:36.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:36.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:36.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:36.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~87s 2023-07-04T23:41:36.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~87s 2023-07-04T23:41:37.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:37.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:37.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:37.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:37.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:37.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:37.209 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:37.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:37.209 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:37.208+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:37.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:37.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:37.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:37.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:37.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:37.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:37.479 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:37.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:37.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:37.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:37.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:37.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:37.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:37.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:37.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:37.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:37.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:37.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:37.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:37.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:37.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:37.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:37.912 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:37.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:38.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:38.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:38.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:38.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:38.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:38.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:38.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:38.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:38.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:38.225+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:38.474 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:38.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:38.474 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:38.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:38.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:38.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:38.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:38.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:38.829 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:38.828+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:38.830 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:38.828+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:38.830 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:38.828+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:38.830 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:38.828+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:38.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:38.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:38.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:38.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:38.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:38.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:38.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:38.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:39.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:39.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:39.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:39.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:39.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:39.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:39.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:39.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:39.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:39.272+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:39.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:39.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:39.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:39.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:39.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:39.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:39.446 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:39.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:39.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:39.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:39.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:39.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:39.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:39.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:39.858 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:39.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:39.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:39.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:39.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:39.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:39.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:39.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:39.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:39.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:40.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:40.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:40.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:40.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:40.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:40.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:40.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:40.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:40.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:40.258+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:40.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:41:40.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:41:40.481 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:40.480+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:40.481 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:40.480+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:40.482 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:40.480+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:40.482 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:40.480+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:40.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:40.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:40.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:40.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:40.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:40.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:40.880 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:40.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:40.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:40.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:40.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:40.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:40.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:40.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:40.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:40.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:41.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:41.234+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:41.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:41.234+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:41.235 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:41.234+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:41.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:41.234+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:41.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:41.234+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:41.368 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:41.366+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1385 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:41.509 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:41.508+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:41.509 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:41.508+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:41.509 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:41.508+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:41.510 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:41.508+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:41.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:41.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:41.899 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:41.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:41.899 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:41.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:41.899 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:41.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:41.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:41.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:41.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:41.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:41.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:41.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:41.910 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:41.909+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:42.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:42.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:42.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:42.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:42.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:42.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:42.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:42.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:42.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:42.264+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:42.505 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:42.504+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:42.505 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:42.504+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:42.505 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:42.504+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:42.505 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:42.504+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:42.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~93s 2023-07-04T23:41:42.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~93s 2023-07-04T23:41:42.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~93s 2023-07-04T23:41:42.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~93s 2023-07-04T23:41:42.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~93s 2023-07-04T23:41:42.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~93s 2023-07-04T23:41:42.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~93s 2023-07-04T23:41:42.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~93s 2023-07-04T23:41:42.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~93s 2023-07-04T23:41:42.871 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:42.870+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:42.871 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:42.870+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:42.871 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:42.870+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:42.871 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:42.870+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:42.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:42.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:42.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:42.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:42.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:42.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:42.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:42.909+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:43.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:43.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:43.259 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:43.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:43.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:43.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:43.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:43.258+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:43.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:43.258+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:43.504 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:43.503+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:43.504 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:43.503+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:43.504 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:43.503+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:43.504 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:43.503+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:43.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:43.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:43.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:43.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:43.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:43.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:43.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:43.857+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:43.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:43.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:43.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:43.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:43.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:43.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:43.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:43.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:44.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:44.302+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:44.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:44.302+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:44.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:44.302+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:44.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:44.302+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:44.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:44.302+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:44.480 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:44.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:44.480 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:44.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:44.480 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:44.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:44.480 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:44.479+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:44.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:44.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:44.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:44.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:44.886 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:44.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:44.886 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:44.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:44.924 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:44.923+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:44.924 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:44.923+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:44.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:44.923+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:44.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:44.923+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:45.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:45.350+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:45.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:45.350+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:45.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:45.350+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:45.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:45.350+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:45.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:45.350+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:45.521 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:45.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:45.521 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:45.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:45.522 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:45.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:45.522 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:45.520+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:45.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:45.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:45.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:45.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:45.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:45.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:45.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:45.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:45.891 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:45.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:45.892 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:45.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:45.892 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:45.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:45.892 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:45.890+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:46.368 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:46.367+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1462 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:46.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:46.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:46.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:46.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:46.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:46.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:46.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:46.380+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:46.381 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:46.380+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:46.538 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:46.537+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:46.538 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:46.537+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:46.538 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:46.537+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:46.538 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:46.537+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:46.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:46.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:46.869 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:46.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:46.869 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:46.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:46.869 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:46.867+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:46.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:46.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:46.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:46.898 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:46.897+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:47.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:47.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:47.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:47.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:47.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:47.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:47.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:47.371+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:47.373 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:47.371+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:47.491 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:47.490+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:47.492 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:47.490+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:47.492 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:47.490+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:47.492 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:47.490+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:47.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:47.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:47.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:47.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:47.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:47.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:47.906 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:47.904+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:47.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:47.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:47.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:47.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:47.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:47.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:47.935 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:47.934+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:48.402 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:48.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:48.402 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:48.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:48.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:48.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:48.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:48.401+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:48.403 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:48.401+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:48.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:48.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:48.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:48.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:48.521 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:48.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:48.521 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:48.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~99s 2023-07-04T23:41:48.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~99s 2023-07-04T23:41:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~99s 2023-07-04T23:41:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~99s 2023-07-04T23:41:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~99s 2023-07-04T23:41:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~99s 2023-07-04T23:41:48.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~99s 2023-07-04T23:41:48.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~99s 2023-07-04T23:41:48.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~99s 2023-07-04T23:41:48.874 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:48.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:48.874 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:48.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:48.874 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:48.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:48.875 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:48.873+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:48.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:48.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:48.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:48.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:48.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:48.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:48.913 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:48.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:49.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:49.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:49.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:49.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:49.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:49.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:49.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:49.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:49.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:49.365+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:49.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:49.542+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:49.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:49.542+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:49.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:49.542+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:49.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:49.542+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:49.849 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:49.848+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:49.849 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:49.848+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:49.850 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:49.848+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:49.850 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:49.848+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:49.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:49.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:49.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:49.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:49.878 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:49.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:49.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:49.877+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:50.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:50.349 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:50.349 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:50.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:50.349 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:50.347+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:50.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:50.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:50.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:50.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:50.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:50.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:50.560 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:50.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:50.844 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:50.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:50.844 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:50.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:50.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:50.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:50.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:50.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:50.876 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:50.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:50.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:50.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:50.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:50.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:50.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:50.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:51.302 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:51.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:51.302 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:51.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:51.302 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:51.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:51.302 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:51.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:51.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:51.301+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:51.369 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:51.368+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1554 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:51.557 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:51.556+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:51.558 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:51.556+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:51.558 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:51.556+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:51.558 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:51.556+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:51.817 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:51.816+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:51.817 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:51.816+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:51.817 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:51.816+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:51.817 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:51.816+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:51.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:51.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:51.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:51.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:51.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:51.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:51.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:51.912+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:52.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:52.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:52.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:52.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:52.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:52.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:52.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:52.301+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:52.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:52.301+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:52.569 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:52.568+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:52.569 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:52.568+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:52.570 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:52.568+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:52.570 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:52.568+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:52.847 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:52.846+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:52.847 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:52.846+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:52.847 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:52.846+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:52.847 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:52.846+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:52.919 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:52.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:52.919 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:52.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:52.920 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:52.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:52.920 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:52.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:53.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:53.275+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:53.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:53.275+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:53.277 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:53.275+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:53.277 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:53.275+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:53.277 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:53.275+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:53.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:53.593+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:53.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:53.593+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:53.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:53.593+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:53.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:53.593+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:53.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:53.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:53.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:53.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:53.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:53.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:53.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:53.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:53.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:53.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:53.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:53.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:53.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:53.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:53.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:53.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:54.288 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:54.287+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:54.289 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:54.287+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:54.289 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:54.287+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:54.289 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:54.287+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:54.289 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:54.287+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:54.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~105s 2023-07-04T23:41:54.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~105s 2023-07-04T23:41:54.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~105s 2023-07-04T23:41:54.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~105s 2023-07-04T23:41:54.637 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:54.636+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:54.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:54.636+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:54.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:54.636+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:54.638 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:54.636+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:54.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:54.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:54.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:54.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:54.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:54.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:54.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:54.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:54.890 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:54.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:54.890 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:54.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:54.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:54.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:54.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:54.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:55.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:55.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:55.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:55.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:55.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:55.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:55.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:55.261+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:55.262 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:55.261+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:55.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:55.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:55.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:55.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:55.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:55.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:55.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:55.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:55.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:55.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:55.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:55.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:55.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:55.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:55.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:55.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:55.923 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:55.923+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:55.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:55.923+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:55.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:55.923+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:55.924 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:55.923+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:56.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:56.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:56.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:56.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:56.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:56.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:56.247 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:56.245+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:56.247 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:56.245+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:56.370 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:41:56.369+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1636 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:41:56.631 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:56.630+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:56.631 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:56.630+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:56.631 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:56.630+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:56.631 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:56.630+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:56.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:56.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:56.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:56.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:56.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:56.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:56.887 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:56.886+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:56.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:56.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:56.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:56.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:56.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:56.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:56.925 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:56.924+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:57.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:57.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:57.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:57.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:57.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:57.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:57.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:57.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:57.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:57.216+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:57.680 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:57.679+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:57.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:57.679+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:57.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:57.679+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:57.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:57.679+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:57.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:57.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:57.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:57.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:57.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:57.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:57.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:57.876+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:57.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:57.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:57.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:57.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:57.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:57.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:57.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:57.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:58.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:58.212+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:58.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:58.212+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:58.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:58.212+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:58.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:58.212+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:58.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:58.212+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:58.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:58.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:58.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:58.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:58.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:58.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:58.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:58.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:58.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:58.898+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:58.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:58.898+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:58.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:58.898+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:58.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:58.898+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:58.972 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:58.972+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:58.973 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:58.972+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:58.973 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:58.972+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:58.973 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:58.972+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:59.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:59.241+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:59.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:59.241+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:59.242 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:59.241+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:41:59.242 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:59.241+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:41:59.242 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:41:59.241+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:41:59.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:59.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:59.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:59.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:41:59.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:59.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:59.693 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:41:59.692+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:41:59.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:59.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:59.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:59.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:59.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:59.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:41:59.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:41:59.862+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:41:59.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:59.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:59.944 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:59.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:41:59.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:59.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:41:59.945 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:41:59.943+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:00.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~110s 2023-07-04T23:42:00.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~110s 2023-07-04T23:42:00.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~110s 2023-07-04T23:42:00.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~110s 2023-07-04T23:42:00.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~110s 2023-07-04T23:42:00.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~110s 2023-07-04T23:42:00.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~110s 2023-07-04T23:42:00.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~110s 2023-07-04T23:42:00.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~110s 2023-07-04T23:42:00.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:00.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:00.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:00.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:00.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:00.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:00.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:00.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:00.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:00.214+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:00.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:00.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:00.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:00.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:00.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:00.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:00.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:00.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:00.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:00.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:00.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:00.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:00.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:00.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:00.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:00.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:00.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:00.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:00.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:00.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:00.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:00.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:00.975 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:00.974+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:01.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:01.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:01.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:01.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:01.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:01.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:01.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:01.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:01.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:01.259+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:01.371 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:01.370+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1712 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:01.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:01.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:01.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:01.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:01.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:01.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:01.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:01.700+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:01.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:01.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:01.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:01.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:01.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:01.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:01.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:01.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:01.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:01.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:01.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:01.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:01.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:01.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:01.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:01.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:02.229 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:02.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:02.229 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:02.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:02.229 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:02.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:02.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:02.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:02.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:02.228+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:02.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:02.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:02.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:02.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:02.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:02.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:02.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:02.705+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:02.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:02.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:02.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:02.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:02.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:02.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:02.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:02.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:03.006 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:03.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:03.006 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:03.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:03.006 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:03.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:03.006 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:03.005+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:03.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:03.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:03.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:03.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:03.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:03.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:03.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:03.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:03.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:03.255+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:03.745 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:03.745+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:03.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:03.745+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:03.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:03.745+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:03.746 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:03.745+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:03.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:03.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:03.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:03.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:03.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:03.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:03.813 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:03.812+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:04.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:04.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:04.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:04.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:04.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:04.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:04.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:04.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:04.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:04.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:04.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:04.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:04.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:04.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:04.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:04.252+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:04.254 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:04.252+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:04.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:04.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:04.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:04.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:04.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:04.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:04.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:04.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:04.840 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:04.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:04.840 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:04.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:04.841 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:04.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:04.841 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:04.840+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:05.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:05.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:05.003 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:05.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.003+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:05.285 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:05.285+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:05.285 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:05.285+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:05.286 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:05.285+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:05.286 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:05.285+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:05.286 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:05.285+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:05.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:05.796+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:05.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:05.796+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:05.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:05.796+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:05.797 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:05.796+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:05.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:05.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:05.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:05.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:05.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:05.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:05.818 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:05.817+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:05.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~116s 2023-07-04T23:42:05.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~116s 2023-07-04T23:42:05.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~116s 2023-07-04T23:42:05.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~116s 2023-07-04T23:42:05.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~116s 2023-07-04T23:42:05.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~116s 2023-07-04T23:42:05.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~116s 2023-07-04T23:42:05.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~116s 2023-07-04T23:42:05.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~116s 2023-07-04T23:42:05.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:05.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:05.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:05.987 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:05.986+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:06.330 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:06.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:06.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:06.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:06.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:06.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:06.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:06.329+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:06.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:06.330+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:06.371 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:06.370+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1802 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:06.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:06.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:06.807 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:06.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:06.807 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:06.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:06.807 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:06.806+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:06.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:06.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:06.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:06.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:06.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:06.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:06.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:06.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:07.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:07.004+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:07.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:07.004+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:07.004 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:07.004+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:07.005 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:07.004+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:07.326 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:07.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:07.326 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:07.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:07.326 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:07.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:07.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:07.325+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:07.327 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:07.325+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:07.811 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:07.810+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:07.811 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:07.810+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:07.811 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:07.810+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:07.811 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:07.810+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:07.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:07.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:07.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:07.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:07.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:07.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:07.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:07.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:08.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:08.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:08.325 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:08.324+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:08.325 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:08.324+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:08.325 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:08.324+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:08.325 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:08.324+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:08.325 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:08.324+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:08.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:08.791+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:08.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:08.791+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:08.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:08.791+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:08.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:08.791+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:08.814 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:08.813+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:08.814 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:08.813+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:08.814 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:08.813+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:08.814 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:08.813+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:09.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:09.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:09.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:09.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:09.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:09.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:09.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:09.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:09.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:09.312+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:09.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:09.312+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:09.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:09.312+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:09.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:09.312+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:09.313 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:09.312+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:09.796 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:09.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:09.797 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:09.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:09.797 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:09.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:09.797 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:09.796+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:09.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:09.834+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:09.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:09.834+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:09.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:09.834+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:09.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:09.834+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:10.030 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:10.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:10.030 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:10.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:10.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:10.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:10.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:10.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:10.293 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:10.292+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:10.293 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:10.292+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:10.293 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:10.292+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:10.293 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:10.292+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:10.293 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:10.292+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:10.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:42:10.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:42:10.783 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:10.782+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:10.783 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:10.782+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:10.783 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:10.782+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:10.783 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:10.782+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:10.803 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:10.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:10.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:10.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:10.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:10.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:10.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:10.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:11.019 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:11.017+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:11.019 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:11.018+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:11.019 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:11.018+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:11.019 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:11.018+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:11.271 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:11.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:11.271 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:11.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:11.271 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:11.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:11.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:11.271+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:11.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:11.271+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:11.372 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:11.371+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1879 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:11.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~122s 2023-07-04T23:42:11.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~122s 2023-07-04T23:42:11.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~122s 2023-07-04T23:42:11.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~122s 2023-07-04T23:42:11.767 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:11.766+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:11.767 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:11.766+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:11.767 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:11.766+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:11.768 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:11.766+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:11.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:11.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:11.835 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:11.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:11.835 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:11.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:11.835 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:11.833+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:12.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:12.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:12.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:12.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:12.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:12.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:12.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:12.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:12.299 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:12.298+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:12.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:12.298+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:12.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:12.298+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:12.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:12.298+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:12.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:12.298+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:12.775 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:12.775+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:12.776 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:12.775+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:12.776 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:12.775+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:12.776 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:12.775+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:12.866 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:12.865+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:12.866 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:12.865+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:12.866 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:12.865+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:12.867 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:12.865+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:13.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:13.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:13.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:13.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:13.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:13.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:13.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:13.047+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:13.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:13.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:13.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:13.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:13.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:13.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:13.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:13.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:13.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:13.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:13.771 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:13.770+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:13.771 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:13.770+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:13.771 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:13.770+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:13.771 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:13.770+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:13.855 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:13.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:13.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:13.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:13.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:13.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:13.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:13.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:14.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:14.057+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:14.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:14.057+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:14.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:14.057+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:14.058 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:14.057+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:14.305 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:14.305+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:14.306 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:14.305+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:14.306 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:14.305+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:14.306 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:14.305+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:14.306 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:14.305+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:14.777 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:14.776+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:14.777 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:14.776+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:14.777 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:14.776+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:14.777 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:14.776+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:14.821 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:14.820+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:14.821 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:14.820+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:14.821 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:14.820+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:14.821 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:14.820+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:15.062 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:15.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:15.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:15.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:15.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:15.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:15.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:15.062+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:15.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:15.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:15.298 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:15.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:15.299 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:15.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:15.299 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:15.297+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:15.299 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:15.297+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:15.808 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:15.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:15.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:15.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:15.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:15.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:15.809 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:15.808+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:15.840 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:15.839+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:15.840 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:15.839+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:15.840 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:15.839+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:15.840 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:15.839+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:16.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:16.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:16.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:16.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:16.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:16.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:16.050 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:16.049+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:16.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:16.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:16.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:16.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:16.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:16.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:16.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:16.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:16.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:16.330+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:16.373 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:16.372+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 1955 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:16.790 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:16.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:16.790 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:16.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:16.791 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:16.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:16.791 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:16.790+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:16.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:16.804+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:16.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:16.804+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:16.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:16.804+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:16.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:16.804+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:17.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:17.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:17.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:17.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:17.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:17.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:17.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:17.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:17.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:17.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:17.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:17.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:17.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:17.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:17.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:17.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:17.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:17.310+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:17.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~128s 2023-07-04T23:42:17.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~128s 2023-07-04T23:42:17.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~128s 2023-07-04T23:42:17.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~128s 2023-07-04T23:42:17.764 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:17.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:17.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:17.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:17.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:17.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:17.765 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:17.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:17.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:17.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:17.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:17.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:17.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:17.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:17.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:17.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:18.075 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:18.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:18.075 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:18.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:18.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:18.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:18.076 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:18.075+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:18.290 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:18.289+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:18.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:18.289+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:18.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:18.289+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:18.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:18.289+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:18.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:18.289+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:18.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:18.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:18.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:18.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:18.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:18.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:18.769 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:18.768+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:18.805 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:18.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:18.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:18.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:18.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:18.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:18.806 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:18.805+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:19.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:19.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:19.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:19.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:19.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:19.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:19.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:19.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:19.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:19.246+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:19.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:19.246+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:19.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:19.246+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:19.247 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:19.246+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:19.247 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:19.246+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:19.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:19.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:19.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:19.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:19.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:19.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:19.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:19.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:19.787+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:20.051 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:20.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:20.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:20.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:20.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:20.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:20.052 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:20.051+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:20.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:20.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:20.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:20.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:20.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:20.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:20.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:20.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:20.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:20.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:20.741 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:20.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:20.742 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:20.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:20.742 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:20.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:20.742 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:20.741+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:20.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:20.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:20.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:20.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:20.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:20.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:20.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:20.759+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:21.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:21.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:21.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:21.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:21.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:21.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:21.086 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:21.086+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:21.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:21.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:21.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:21.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:21.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:21.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:21.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:21.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:21.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:21.197+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:21.374 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:21.373+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2040 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:21.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:21.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:21.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:21.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:21.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:21.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:21.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:21.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:21.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:21.758+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:21.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:21.758+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:21.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:21.758+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:21.758 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:21.758+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:22.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:22.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:22.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:22.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:22.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:22.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:22.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:22.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:22.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:22.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:22.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:22.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:22.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:22.230+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:22.758 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:22.757+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:22.758 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:22.757+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:22.758 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:22.757+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:22.758 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:22.757+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:22.788 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:22.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:22.788 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:22.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:22.788 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:22.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:22.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:22.788+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:23.108 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:23.108+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:23.109 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:23.108+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:23.109 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:23.108+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:23.109 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:23.108+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:23.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:23.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:23.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:23.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:23.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:23.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:23.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:23.279+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:23.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:23.279+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:23.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~134s 2023-07-04T23:42:23.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~134s 2023-07-04T23:42:23.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~134s 2023-07-04T23:42:23.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~134s 2023-07-04T23:42:23.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~134s 2023-07-04T23:42:23.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~134s 2023-07-04T23:42:23.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~134s 2023-07-04T23:42:23.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~134s 2023-07-04T23:42:23.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~134s 2023-07-04T23:42:23.735 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:23.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:23.735 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:23.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:23.735 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:23.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:23.736 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:23.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:23.765 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:23.765+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:23.765 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:23.765+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:23.765 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:23.765+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:23.765 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:23.765+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:24.122 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:24.122+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:24.122 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:24.122+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:24.122 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:24.122+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:24.122 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:24.122+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:24.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:24.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:24.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:24.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:24.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:24.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:24.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:24.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:24.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:24.273+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:24.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:24.773+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:24.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:24.773+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:24.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:24.773+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:24.773 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:24.773+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:24.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:24.790+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:24.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:24.790+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:24.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:24.790+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:24.791 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:24.790+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:25.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:25.093+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:25.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:25.093+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:25.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:25.093+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:25.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:25.093+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:25.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:25.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:25.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:25.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:25.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:25.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:25.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:25.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:25.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:25.310+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:25.784 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:25.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:25.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:25.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:25.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:25.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:25.785 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:25.784+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:25.797 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:25.797+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:25.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:25.797+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:25.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:25.797+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:25.798 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:25.797+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:26.062 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:26.061+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:26.062 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:26.061+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:26.062 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:26.061+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:26.063 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:26.061+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:26.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:26.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:26.323 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:26.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:26.323 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:26.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:26.323 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:26.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:26.323 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:26.322+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:26.375 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:26.374+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2120 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:26.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:26.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:26.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:26.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:26.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:26.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:26.741 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:26.740+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:26.800 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:26.799+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:26.800 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:26.799+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:26.800 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:26.799+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:26.801 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:26.799+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:27.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:27.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:27.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:27.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:27.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:27.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:27.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:27.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:27.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:27.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:27.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:27.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:27.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:27.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:27.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:27.274+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:27.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:27.274+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:27.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:27.721+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:27.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:27.721+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:27.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:27.721+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:27.722 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:27.721+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:27.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:27.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:27.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:27.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:27.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:27.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:27.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:27.828+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:28.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:28.069+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:28.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:28.069+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:28.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:28.069+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:28.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:28.069+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:28.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:28.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:28.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:28.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:28.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:28.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:28.260 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:28.259+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:28.261 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:28.259+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:28.757 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:28.756+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:28.757 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:28.756+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:28.757 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:28.756+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:28.757 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:28.756+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:28.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:28.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:28.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:28.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:28.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:28.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:28.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:28.871+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:29.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:29.035+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:29.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:29.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:29.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:29.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:29.037 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:29.036+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:29.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~139s 2023-07-04T23:42:29.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~139s 2023-07-04T23:42:29.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~139s 2023-07-04T23:42:29.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~139s 2023-07-04T23:42:29.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:29.266+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:29.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:29.266+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:29.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:29.266+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:29.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:29.266+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:29.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:29.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:29.749 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:29.748+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:29.749 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:29.748+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:29.749 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:29.748+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:29.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:29.748+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:29.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:29.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:29.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:29.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:29.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:29.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:29.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:29.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:30.040 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:30.039+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:30.040 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:30.039+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:30.040 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:30.039+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:30.040 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:30.039+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:30.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:30.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:30.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:30.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:30.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:30.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:30.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:30.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:30.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:30.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:30.700 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:30.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:30.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:30.701 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:30.699+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:30.830 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:30.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:30.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:30.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:30.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:30.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:30.831 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:30.829+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:31.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:31.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:31.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:31.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:31.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:31.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:31.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:31.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:31.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:31.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:31.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:31.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:31.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:31.216+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:31.376 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:31.375+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2195 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:31.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:31.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:31.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:31.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:31.697 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:31.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:31.698 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:31.696+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:31.850 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:31.849+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:31.851 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:31.849+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:31.851 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:31.849+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:31.851 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:31.849+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:31.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:31.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:31.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:31.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:31.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:32.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:32.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:32.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:32.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:32.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:32.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:32.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:32.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:32.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:32.219+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:32.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:32.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:32.674 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:32.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:32.674 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:32.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:32.674 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:32.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:32.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:32.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:32.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:32.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:32.886 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:32.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:32.886 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:32.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:32.951 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:32.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:32.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:32.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:32.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:32.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:32.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:32.950+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:33.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:33.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:33.230+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:33.719 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:33.718+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:33.719 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:33.718+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:33.719 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:33.718+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:33.720 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:33.718+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:33.933 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:33.932+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:33.933 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:33.932+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:33.933 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:33.932+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:33.933 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:33.932+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:33.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:33.952+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:33.953 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:33.952+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:33.953 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:33.952+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:33.953 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:33.952+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:34.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:34.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:34.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:34.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:34.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:34.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:34.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:34.256+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:34.257 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:34.256+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:34.686 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:34.686+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:34.687 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:34.686+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:34.687 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:34.686+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:34.687 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:34.686+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:34.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:34.944+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:34.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:34.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:34.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:34.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:34.946 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:34.945+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:34.954 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:34.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:34.954 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:34.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:34.954 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:34.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:34.954 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:34.953+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~145s 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~145s 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~145s 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~145s 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~145s 2023-07-04T23:42:34.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~145s 2023-07-04T23:42:34.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~145s 2023-07-04T23:42:34.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~145s 2023-07-04T23:42:34.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~145s 2023-07-04T23:42:35.211 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:35.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:35.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:35.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:35.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:35.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:35.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:35.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:35.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:35.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:35.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:35.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:35.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:35.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:35.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:35.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:35.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:35.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:35.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:35.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:35.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:35.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:35.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:35.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:35.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:35.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:35.997 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:35.996+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:35.998 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:35.996+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:35.998 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:35.996+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:35.998 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:35.996+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:36.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:36.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:36.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:36.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:36.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:36.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:36.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:36.203+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:36.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:36.203+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:36.377 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:36.376+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2280 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:36.743 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:36.742+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:36.743 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:36.742+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:36.743 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:36.742+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:36.743 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:36.742+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:36.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:36.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:36.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:36.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:36.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:36.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:36.951 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:36.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:37.001 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.000+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:37.002 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.000+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:37.002 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.000+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:37.002 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.000+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:37.218 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:37.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:37.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:37.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:37.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:37.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:37.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:37.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:37.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:37.218+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:37.763 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:37.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:37.764 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:37.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:37.764 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:37.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:37.764 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:37.762+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:37.904 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:37.902+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:37.904 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:37.903+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:37.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:37.903+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:37.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:37.903+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:37.994 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:37.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:37.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:37.995 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:37.993+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:38.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:38.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:38.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:38.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:38.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:38.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:38.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:38.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:38.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:38.221+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:38.804 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:38.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:38.805 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:38.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:38.805 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:38.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:38.805 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:38.803+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:38.867 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:38.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:38.867 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:38.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:38.867 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:38.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:38.867 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:38.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:39.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:39.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:39.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:39.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.012+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:39.254 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:39.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:39.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:39.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:39.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:39.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:39.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:39.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:39.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:39.254+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:39.842 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:39.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:39.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:39.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:39.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:39.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:39.843 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:39.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:39.867 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:39.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:39.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:39.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:39.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:39.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:39.868 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:39.866+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:39.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:39.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:39.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:39.971 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:39.970+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:40.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:40.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:40.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:40.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:40.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:40.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:40.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:40.254+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:40.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:40.254+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:40.588 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:42:40.591 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:42:40.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~151s 2023-07-04T23:42:40.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~151s 2023-07-04T23:42:40.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~151s 2023-07-04T23:42:40.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~151s 2023-07-04T23:42:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~151s 2023-07-04T23:42:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~151s 2023-07-04T23:42:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~151s 2023-07-04T23:42:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~151s 2023-07-04T23:42:40.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~151s 2023-07-04T23:42:40.820 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:40.819+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:40.820 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:40.819+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:40.820 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:40.819+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:40.820 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:40.819+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:40.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:40.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:40.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:40.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:40.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:40.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:40.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:40.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:40.962 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:40.961+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:41.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:41.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:41.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:41.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:41.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:41.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:41.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:41.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:41.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:41.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:41.377 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:41.376+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2358 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:41.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:41.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:41.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:41.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:41.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:41.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:41.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:41.794+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:41.864 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:41.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:41.864 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:41.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:41.864 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:41.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:41.864 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:41.863+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:41.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:41.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:41.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:41.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:41.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:41.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:41.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:41.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:42.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:42.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:42.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:42.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:42.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:42.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:42.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:42.219+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:42.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:42.219+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:42.817 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:42.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:42.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:42.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:42.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:42.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:42.818 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:42.817+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:42.827 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:42.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:42.827 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:42.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:42.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:42.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:42.828 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:42.827+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:43.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:43.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:43.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:43.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:43.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:43.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:43.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:43.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:43.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:43.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:43.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:43.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:43.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:43.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:43.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:43.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:43.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:43.222+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:43.787 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:43.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:43.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:43.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:43.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:43.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:43.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:43.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:43.836 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:43.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:43.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:43.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:43.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:43.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:43.837 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:43.836+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:44.040 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:44.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:44.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:44.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:44.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:44.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:44.041 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:44.040+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:44.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:44.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:44.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:44.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:44.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:44.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:44.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:44.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:44.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:44.214+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:44.821 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:44.821+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:44.822 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:44.821+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:44.822 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:44.821+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:44.822 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:44.821+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:44.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:44.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:44.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:44.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:44.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:44.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:44.880 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:44.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:45.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:45.090+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:45.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:45.090+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:45.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:45.090+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:45.091 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:45.090+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:45.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:45.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:45.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:45.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:45.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:45.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:45.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:45.222+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:45.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:45.222+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:45.857 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:45.857+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:45.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:45.857+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:45.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:45.857+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:45.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:45.857+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:45.895 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:45.894+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:45.895 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:45.894+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:45.895 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:45.894+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:45.895 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:45.894+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:46.100 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:46.099+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:46.101 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:46.099+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:46.101 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:46.099+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:46.101 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:46.099+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:46.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:46.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:46.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:46.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:46.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:46.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:46.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:46.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:46.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:46.233+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:46.378 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:46.377+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2436 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~157s 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~157s 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~157s 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~157s 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~157s 2023-07-04T23:42:46.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~157s 2023-07-04T23:42:46.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~157s 2023-07-04T23:42:46.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~157s 2023-07-04T23:42:46.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~157s 2023-07-04T23:42:46.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:46.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:46.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:46.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:46.856 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:46.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:46.857 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:46.855+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:46.872 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:46.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:46.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:46.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:46.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:46.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:46.873 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:46.872+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:47.080 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:47.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:47.080 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:47.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:47.080 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:47.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:47.080 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:47.079+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:47.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:47.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:47.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:47.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:47.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:47.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:47.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:47.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:47.245 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:47.243+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:47.883 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:47.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:47.883 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:47.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:47.883 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:47.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:47.883 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:47.882+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:47.898 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:47.897+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:47.898 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:47.897+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:47.898 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:47.897+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:47.898 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:47.897+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:48.056 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:48.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:48.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:48.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:48.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:48.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:48.057 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:48.056+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:48.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:48.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:48.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:48.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:48.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:48.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:48.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:48.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:48.266 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:48.264+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:48.919 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:48.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:48.919 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:48.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:48.920 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:48.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:48.920 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:48.918+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:48.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:48.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:48.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:48.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:48.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:48.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:48.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:48.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:49.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:49.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:49.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:49.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:49.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:49.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:49.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:49.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:49.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:49.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:49.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:49.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:49.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:49.225+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:49.916 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:49.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:49.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:49.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:49.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:49.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:49.917 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:49.915+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:49.917 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:49.916+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:49.917 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:49.916+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:49.918 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:49.916+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:49.918 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:49.916+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:49.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:49.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:49.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:49.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:49.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:50.270 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:50.269+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:50.270 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:50.269+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:50.270 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:50.269+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:50.270 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:50.269+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:50.270 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:50.269+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:50.926 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:50.925+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:50.926 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:50.925+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:50.927 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:50.925+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:50.927 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:50.925+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:50.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:50.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:50.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:50.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:50.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:50.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:50.941 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:50.940+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:50.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:50.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:50.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:50.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:50.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:50.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:50.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:50.951+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:51.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:51.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:51.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:51.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:51.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:51.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:51.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:51.233+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:51.234 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:51.233+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:51.379 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:51.378+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2523 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:51.882 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:51.882+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:51.883 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:51.882+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:51.883 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:51.882+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:51.883 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:51.882+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:51.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:51.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:51.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:51.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:51.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:51.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:51.986 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:51.984+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:52.000 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:51.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:52.000 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:51.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:52.000 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:51.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:52.000 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:51.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:52.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:52.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:52.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:52.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:52.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:52.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:52.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:52.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:52.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:52.248+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:52.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~163s 2023-07-04T23:42:52.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~163s 2023-07-04T23:42:52.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~163s 2023-07-04T23:42:52.846 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:52.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:52.846 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:52.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:52.846 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:52.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:52.846 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:52.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:53.005 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.004+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.005 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.004+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.005 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.004+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.005 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.004+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:53.021 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:53.021 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:53.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:53.022 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.021+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:53.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:53.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:53.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:53.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:53.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:53.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:53.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:53.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:53.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:53.242+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:53.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:53.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:53.798 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:53.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:53.799 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:53.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:53.799 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:53.798+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:53.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:53.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:53.963+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:53.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:53.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:53.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:53.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:53.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:54.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:54.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:54.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:54.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:54.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:54.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:54.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:54.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:54.239 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:54.238+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:54.829 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:54.828+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:54.830 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:54.828+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:54.830 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:54.828+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:54.830 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:54.828+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:54.982 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:54.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:54.982 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:54.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:54.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:54.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:54.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:54.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:55.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:55.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:55.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:55.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:55.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:55.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:55.038 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:55.037+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:55.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:55.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:55.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:55.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:55.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:55.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:55.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:55.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:55.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:55.198+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:55.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:55.807+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:55.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:55.807+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:55.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:55.807+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:55.808 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:55.807+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:56.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:56.030+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:56.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:56.030+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:56.031 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:56.030+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:56.031 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:56.030+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:56.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:56.060+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:56.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:56.060+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:56.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:56.060+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:56.061 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:56.060+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:56.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:56.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:56.248 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:56.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:56.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:56.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:56.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:56.248+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:56.249 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:56.248+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:56.380 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:42:56.379+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2601 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:42:56.850 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:56.850+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:56.850 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:56.850+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:56.850 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:56.850+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:56.850 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:56.850+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:57.013 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:57.013+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:57.014 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:57.013+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:57.014 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:57.013+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:57.014 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:57.013+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:57.049 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:57.049+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:57.050 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:57.049+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:57.050 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:57.049+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:57.050 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:57.049+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:57.255 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:57.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:57.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:57.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:57.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:57.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:57.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:57.255+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:57.256 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:57.255+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:57.827 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:57.827+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:57.827 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:57.827+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:57.827 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:57.827+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:57.828 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:57.827+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:58.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:58.045+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:58.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:58.045+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:58.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:58.045+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:58.046 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:58.045+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:58.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:58.051+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:58.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:58.051+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:58.052 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:58.051+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:58.053 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:58.051+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:58.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~168s 2023-07-04T23:42:58.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~168s 2023-07-04T23:42:58.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~168s 2023-07-04T23:42:58.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~168s 2023-07-04T23:42:58.304 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:58.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:58.305 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:58.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:58.305 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:58.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:58.305 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:58.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:58.305 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:58.304+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:58.842 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:58.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:58.842 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:58.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:58.842 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:58.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:58.842 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:58.841+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:59.069 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:59.068+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:59.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:59.068+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:59.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:59.068+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:42:59.070 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:42:59.068+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:42:59.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:59.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:59.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:59.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:59.099 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:59.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:42:59.099 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:42:59.098+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:42:59.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:59.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:59.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:59.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:59.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:59.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:42:59.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:59.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:42:59.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:42:59.347+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:42:59.837 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:59.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:59.837 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:59.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:42:59.837 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:59.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:42:59.838 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:42:59.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:00.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:00.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:00.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:00.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:00.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:00.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:00.088 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:00.087+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:00.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:00.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:00.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:00.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:00.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:00.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:00.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:00.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:00.303 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:00.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:00.304 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:00.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:00.304 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:00.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:00.304 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:00.303+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:00.304 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:00.303+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:00.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:00.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:00.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:00.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:00.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:00.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:00.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:00.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:01.114 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:01.114+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:01.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:01.114+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:01.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:01.114+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:01.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:01.114+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:01.136 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:01.135+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:01.136 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:01.135+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:01.136 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:01.135+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:01.136 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:01.135+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:01.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:01.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:01.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:01.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:01.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:01.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:01.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:01.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:01.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:01.263+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:01.381 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:01.380+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2678 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:01.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:01.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:01.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:01.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:01.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:01.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:01.776 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:01.775+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:02.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:02.117+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:02.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:02.117+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:02.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:02.117+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:02.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:02.117+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:02.167 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:02.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:02.167 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:02.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:02.167 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:02.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:02.167 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:02.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:02.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:02.291+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:02.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:02.291+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:02.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:02.291+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:02.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:02.291+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:02.291 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:02.291+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:02.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:02.750+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:02.750 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:02.750+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:02.751 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:02.750+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:02.751 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:02.750+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:03.166 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:03.165+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:03.166 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:03.165+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:03.166 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:03.165+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:03.166 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:03.165+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:03.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:03.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:03.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:03.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:03.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:03.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:03.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:03.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:03.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:03.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:03.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:03.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:03.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:03.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:03.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:03.264+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:03.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:03.264+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:03.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:03.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:03.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:03.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:03.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:03.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:03.728 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:03.727+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:04.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~174s 2023-07-04T23:43:04.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~174s 2023-07-04T23:43:04.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~174s 2023-07-04T23:43:04.133 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:04.132+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:04.133 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:04.132+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:04.133 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:04.132+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:04.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:04.132+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:04.185 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:04.185+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:04.185 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:04.185+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:04.186 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:04.185+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:04.186 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:04.185+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:04.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:04.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:04.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:04.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:04.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:04.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:04.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:04.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:04.223 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:04.221+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:04.682 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:04.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:04.682 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:04.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:04.682 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:04.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:04.682 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:04.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:05.105 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:05.104+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:05.106 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:05.104+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:05.106 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:05.104+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:05.106 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:05.104+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:05.192 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:05.191+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:05.192 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:05.191+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:05.192 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:05.191+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:05.192 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:05.191+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:05.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:05.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:05.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:05.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:05.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:05.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:05.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:05.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:05.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:05.204+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:05.665 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:05.664+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:05.665 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:05.664+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:05.665 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:05.664+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:05.665 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:05.664+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:06.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:06.109+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:06.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:06.109+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:06.110 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:06.109+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:06.110 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:06.109+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:06.143 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:06.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:06.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:06.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:06.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:06.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:06.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:06.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:06.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:06.201+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:06.202 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:06.201+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:06.202 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:06.201+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:06.202 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:06.201+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:06.202 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:06.201+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:06.382 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:06.381+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2766 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:06.672 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:06.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:06.672 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:06.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:06.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:06.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:06.673 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:06.672+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:07.142 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:07.141+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:07.142 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:07.141+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:07.142 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:07.141+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:07.143 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:07.141+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:07.149 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:07.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:07.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:07.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:07.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:07.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:07.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:07.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:07.187 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:07.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:07.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:07.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:07.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:07.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:07.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:07.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:07.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:07.187+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:07.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:07.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:07.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:07.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:07.681 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:07.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:07.682 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:07.681+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:08.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:08.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:08.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:08.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:08.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:08.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:08.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:08.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:08.137 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:08.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:08.137 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:08.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:08.137 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:08.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:08.137 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:08.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:08.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:08.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:08.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:08.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:08.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:08.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:08.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:08.215+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:08.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:08.215+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:08.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:08.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:08.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:08.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:08.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:08.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:08.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:08.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:09.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:09.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:09.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:09.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:09.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:09.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:09.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:09.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:09.160 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:09.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:09.160 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:09.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:09.160 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:09.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:09.160 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:09.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:09.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:09.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:09.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:09.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:09.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:09.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:09.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:09.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:09.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:09.206+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:09.635 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:09.635+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:09.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:09.635+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:09.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:09.635+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:09.636 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:09.635+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:09.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2023-07-04T23:43:09.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~180s 2023-07-04T23:43:09.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2023-07-04T23:43:09.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~180s 2023-07-04T23:43:09.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~180s 2023-07-04T23:43:09.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~180s 2023-07-04T23:43:09.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2023-07-04T23:43:09.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~180s 2023-07-04T23:43:09.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~180s 2023-07-04T23:43:10.082 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:10.083+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:10.083 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:10.083+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:10.083 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:10.083+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:10.083 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:10.083+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:10.120 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:10.121+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:10.121+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:10.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:10.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:10.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:10.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:10.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:10.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:10.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:10.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:10.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:10.192+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:10.625 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:10.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:10.626 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:10.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:10.626 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:10.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:10.626 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:10.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:10.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:43:10.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:43:11.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:11.061+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:11.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:11.061+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:11.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:11.061+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:11.062 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:11.061+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:11.158 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:11.158+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:11.158 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:11.158+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:11.158 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:11.158+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:11.159 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:11.158+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:11.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:11.179+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:11.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:11.179+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:11.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:11.179+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:11.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:11.179+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:11.179 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:11.179+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:11.383 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:11.383+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2844 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:11.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:11.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:11.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:11.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:11.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:11.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:11.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:11.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:12.028 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:12.028+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:12.028 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:12.028+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:12.029 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:12.028+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:12.029 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:12.028+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:12.149 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:12.149+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:12.149 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:12.149+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:12.149 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:12.149+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:12.149 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:12.149+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:12.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:12.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:12.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:12.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:12.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:12.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:12.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:12.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:12.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:12.213+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:12.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:12.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:12.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:12.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:12.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:12.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:12.677 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:12.676+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:13.073 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:13.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:13.073 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:13.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:13.073 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:13.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:13.073 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:13.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:13.164 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:13.163+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:13.164 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:13.163+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:13.164 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:13.163+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:13.164 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:13.163+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:13.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:13.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:13.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:13.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:13.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:13.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:13.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:13.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:13.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:13.173+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:13.707 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:13.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:13.707 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:13.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:13.707 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:13.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:13.707 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:13.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:14.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:14.089+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:14.090 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:14.089+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:14.090 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:14.089+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:14.090 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:14.089+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:14.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:14.184+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:14.185 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:14.184+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:14.185 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:14.184+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:14.185 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:14.184+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:14.200 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:14.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:14.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:14.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:14.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:14.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:14.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:14.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:14.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:14.200+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:14.695 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:14.694+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:14.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:14.694+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:14.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:14.694+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:14.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:14.694+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:15.082 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:15.081+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:15.082 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:15.081+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:15.082 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:15.081+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:15.082 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:15.081+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:15.213 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:15.212+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:15.213 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:15.212+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:15.214 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:15.212+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:15.214 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:15.212+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:15.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:15.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:15.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:15.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:15.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:15.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:15.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:15.225+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:15.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:15.226+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:15.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~186s 2023-07-04T23:43:15.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~186s 2023-07-04T23:43:15.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~186s 2023-07-04T23:43:15.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~186s 2023-07-04T23:43:15.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~186s 2023-07-04T23:43:15.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~186s 2023-07-04T23:43:15.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~186s 2023-07-04T23:43:15.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~186s 2023-07-04T23:43:15.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~186s 2023-07-04T23:43:15.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:15.660+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:15.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:15.660+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:15.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:15.660+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:15.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:15.660+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:16.092 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:16.091+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:16.092 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:16.091+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:16.092 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:16.091+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:16.092 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:16.091+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:16.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:16.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:16.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:16.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:16.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:16.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:16.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:16.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:16.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:16.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:16.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:16.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:16.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:16.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:16.263 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:16.262+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:16.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:16.262+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:16.383 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:16.382+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 2919 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:16.709 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:16.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:16.709 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:16.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:16.709 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:16.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:16.709 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:16.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:17.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:17.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:17.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:17.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:17.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:17.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:17.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:17.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:17.199 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:17.198+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:17.199 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:17.198+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:17.199 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:17.198+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:17.199 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:17.198+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:17.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:17.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:17.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:17.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:17.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:17.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:17.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:17.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:17.280 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:17.278+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:17.710 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:17.709+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:17.710 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:17.709+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:17.710 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:17.709+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:17.710 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:17.709+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:18.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:18.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:18.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:18.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:18.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:18.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:18.120 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:18.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:18.235 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:18.234+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:18.235 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:18.234+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:18.235 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:18.234+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:18.235 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:18.234+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:18.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:18.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:18.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:18.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:18.268 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:18.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:18.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:18.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:18.269 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:18.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:18.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:18.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:18.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:18.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:18.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:18.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:18.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:18.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:19.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:19.094+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:19.095 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:19.094+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:19.096 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:19.094+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:19.096 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:19.094+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:19.214 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:19.213+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:19.214 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:19.213+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:19.215 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:19.213+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:19.215 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:19.213+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:19.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:19.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:19.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:19.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:19.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:19.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:19.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:19.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:19.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:19.218+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:19.695 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:19.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:19.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:19.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:19.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:19.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:19.696 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:19.695+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:20.121 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:20.120+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:20.121 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:20.120+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:20.121 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:20.120+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:20.122 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:20.120+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:20.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:20.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:20.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:20.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:20.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:20.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:20.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:20.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:20.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:20.207+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:20.248 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:20.247+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:20.248 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:20.247+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:20.248 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:20.247+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:20.248 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:20.247+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:20.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:20.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:20.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:20.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:20.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:20.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:20.715 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:20.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:21.160 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:21.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:21.160 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:21.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:21.160 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:21.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:21.160 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:21.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:21.209 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:21.208+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:21.209 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:21.208+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:21.209 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:21.208+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:21.209 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:21.208+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:21.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:21.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:21.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:21.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:21.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:21.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:21.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:21.221+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:21.222 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:21.221+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:21.384 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:21.383+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3005 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:21.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~192s 2023-07-04T23:43:21.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~192s 2023-07-04T23:43:21.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~192s 2023-07-04T23:43:21.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:21.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:21.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:21.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:21.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:21.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:21.692 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:21.691+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:22.194 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:22.194+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:22.195 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:22.194+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:22.195 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:22.194+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:22.195 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:22.194+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:22.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:22.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:22.225 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:22.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:22.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:22.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:22.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:22.224+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:22.226 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:22.224+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:22.237 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:22.237+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:22.237 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:22.237+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:22.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:22.237+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:22.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:22.237+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:22.660 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:22.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:22.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:22.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:22.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:22.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:22.661 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:22.659+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:23.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:23.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:23.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:23.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:23.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:23.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:23.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:23.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:23.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:23.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:23.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:23.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:23.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:23.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:23.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:23.207+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:23.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:23.207+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:23.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:23.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:23.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:23.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:23.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:23.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:23.247 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:23.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:23.662 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:23.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:23.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:23.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:23.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:23.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:23.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:23.662+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:24.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:24.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:24.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:24.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:24.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:24.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:24.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:24.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:24.161 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:24.160+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:24.161 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:24.160+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:24.161 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:24.160+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:24.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:24.160+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:24.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:24.161+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:24.281 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:24.280+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:24.314 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:24.280+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:24.314 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:24.280+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:24.315 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:24.280+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:24.634 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:24.633+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:24.634 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:24.633+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:24.634 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:24.633+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:24.635 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:24.633+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:25.129 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:25.129+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:25.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:25.129+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:25.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:25.129+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:25.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:25.129+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:25.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:25.129+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:25.147 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:25.147+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:25.147 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:25.147+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:25.148 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:25.147+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:25.148 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:25.147+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:25.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:25.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:25.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:25.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:25.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:25.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:25.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:25.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:25.591 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:25.591+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:25.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:25.591+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:25.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:25.591+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:25.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:25.591+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:26.098 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:26.097+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:26.098 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:26.097+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:26.098 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:26.097+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:26.099 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:26.097+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:26.099 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:26.097+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:26.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:26.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:26.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:26.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:26.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:26.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:26.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:26.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:26.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:26.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:26.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:26.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:26.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:26.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:26.246 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:26.245+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:26.385 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:26.384+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3084 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:26.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:26.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:26.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:26.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:26.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:26.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:26.559 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:26.558+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:27.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:27.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:27.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:27.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:27.053 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:27.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:27.053 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:27.051+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:27.053 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:27.051+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:27.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:27.197+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:27.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:27.197+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:27.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:27.197+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:27.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:27.197+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:27.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~198s 2023-07-04T23:43:27.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~198s 2023-07-04T23:43:27.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~198s 2023-07-04T23:43:27.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~198s 2023-07-04T23:43:27.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~198s 2023-07-04T23:43:27.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~198s 2023-07-04T23:43:27.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~198s 2023-07-04T23:43:27.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~198s 2023-07-04T23:43:27.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~198s 2023-07-04T23:43:27.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:27.283+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:27.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:27.283+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:27.284 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:27.283+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:27.284 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:27.283+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:27.586 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:27.586+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:27.587 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:27.586+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:27.587 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:27.586+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:27.587 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:27.586+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:28.035 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:28.034+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:28.035 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:28.034+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:28.035 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:28.034+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:28.036 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:28.034+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:28.036 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:28.034+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:28.237 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:28.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:28.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:28.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:28.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:28.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:28.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:28.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:28.240 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:28.239+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:28.240 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:28.239+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:28.240 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:28.239+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:28.240 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:28.239+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:28.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:28.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:28.623 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:28.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:28.623 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:28.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:28.623 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:28.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:29.030 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:29.029+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:29.030 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:29.029+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:29.030 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:29.029+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:29.030 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:29.029+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:29.031 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:29.029+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:29.200 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:29.199+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:29.200 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:29.199+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:29.200 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:29.199+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:29.200 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:29.199+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:29.263 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:29.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:29.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:29.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:29.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:29.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:29.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:29.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:29.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:29.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:29.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:29.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:29.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:29.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:29.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:29.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:30.072 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:30.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:30.072 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:30.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:30.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:30.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:30.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:30.071+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:30.073 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:30.071+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:30.214 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:30.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:30.214 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:30.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:30.214 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:30.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:30.214 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:30.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:30.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:30.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:30.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:30.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:30.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:30.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:30.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:30.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:30.609 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:30.609+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:30.610 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:30.609+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:30.610 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:30.609+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:30.610 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:30.609+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:31.061 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:31.061+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:31.062 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:31.061+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:31.062 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:31.061+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:31.062 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:31.061+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:31.062 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:31.061+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:31.214+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:31.214+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:31.214+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:31.214+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:31.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:31.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:31.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:31.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:31.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:31.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:31.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:31.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:31.386 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:31.385+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3160 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:31.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:31.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:31.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:31.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:31.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:31.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:31.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:31.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:32.017 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:32.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:32.018 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:32.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:32.018 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:32.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:32.018 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:32.016+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:32.018 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:32.016+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:32.218 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:32.217+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:32.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:32.217+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:32.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:32.217+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:32.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:32.217+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:32.243 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:32.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:32.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:32.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:32.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:32.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:32.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:32.243+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:32.628 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:32.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:32.629 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:32.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:32.629 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:32.627+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:32.629 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:32.628+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:33.037 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:33.036+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:33.037 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:33.036+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:33.037 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:33.036+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:33.037 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:33.036+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:33.037 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:33.036+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~203s 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~203s 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~203s 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~203s 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~203s 2023-07-04T23:43:33.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~203s 2023-07-04T23:43:33.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~203s 2023-07-04T23:43:33.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~203s 2023-07-04T23:43:33.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~203s 2023-07-04T23:43:33.207 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:33.206+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:33.207 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:33.206+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:33.207 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:33.206+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:33.207 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:33.206+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:33.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:33.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:33.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:33.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:33.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:33.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:33.283 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:33.282+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:33.603 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:33.603+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:33.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:33.603+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:33.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:33.603+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:33.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:33.603+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:34.057 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:34.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:34.057 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:34.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:34.057 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:34.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:34.058 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:34.056+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:34.058 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:34.056+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:34.221 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:34.221+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:34.221 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:34.221+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:34.222 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:34.221+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:34.222 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:34.221+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:34.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:34.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:34.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:34.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:34.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:34.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:34.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:34.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:34.571 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:34.570+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:34.571 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:34.570+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:34.571 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:34.570+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:34.571 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:34.570+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:35.019 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.018+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.019 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.018+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.019 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.018+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.019 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.018+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:35.019 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.018+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:35.267 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:35.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:35.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:35.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:35.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:35.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:35.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:35.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:35.293 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:35.293+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:35.293 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:35.293+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:35.294 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:35.293+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:35.294 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:35.293+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:35.619 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:35.618+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:35.619 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:35.618+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:35.619 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:35.618+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:35.619 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:35.618+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:35.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.987+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.987+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.987+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:35.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.987+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:35.988 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:35.987+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:36.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:36.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:36.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:36.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:36.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:36.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:36.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:36.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:36.275 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:36.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:36.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:36.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:36.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:36.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:36.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:36.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:36.387 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:36.386+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3245 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:36.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:36.592+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:36.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:36.592+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:36.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:36.592+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:36.592 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:36.592+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:36.942 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:36.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:36.943 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:36.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:36.943 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:36.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:36.943 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:36.941+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:36.943 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:36.941+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:37.226 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:37.226+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:37.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:37.226+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:37.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:37.226+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:37.227 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:37.226+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:37.288 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:37.287+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:37.288 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:37.287+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:37.288 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:37.287+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:37.288 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:37.287+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:37.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:37.621+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:37.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:37.621+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:37.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:37.621+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:37.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:37.621+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:37.946 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:37.946+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:37.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:37.946+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:37.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:37.946+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:37.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:37.946+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:37.947 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:37.946+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:38.259 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:38.258+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:38.259 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:38.258+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:38.259 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:38.258+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:38.259 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:38.258+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:38.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:38.311+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:38.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:38.311+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:38.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:38.311+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:38.312 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:38.311+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:38.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:38.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:38.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:38.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:38.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:38.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:38.645 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:38.645+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~209s 2023-07-04T23:43:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~209s 2023-07-04T23:43:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~209s 2023-07-04T23:43:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~209s 2023-07-04T23:43:38.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~209s 2023-07-04T23:43:38.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~209s 2023-07-04T23:43:38.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~209s 2023-07-04T23:43:38.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~209s 2023-07-04T23:43:38.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~209s 2023-07-04T23:43:38.960 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:38.959+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:38.960 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:38.959+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:38.960 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:38.959+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:38.960 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:38.959+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:38.960 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:38.959+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:39.285 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:39.285+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:39.285 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:39.285+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:39.286 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:39.285+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:39.286 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:39.285+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:39.343 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:39.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:39.343 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:39.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:39.343 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:39.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:39.343 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:39.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:39.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:39.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:39.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:39.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:39.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:39.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:39.669 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:39.669+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:39.944 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:39.944+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:39.945 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:39.944+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:39.945 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:39.944+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:39.945 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:39.944+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:39.945 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:39.944+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:40.313 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:40.312+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:40.313 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:40.312+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:40.313 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:40.312+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:40.314 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:40.312+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:40.347 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:40.346+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:40.347 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:40.346+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:40.348 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:40.346+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:40.348 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:40.346+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:40.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:40.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:40.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:40.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:40.627 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:40.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:40.628 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:40.626+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:40.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:43:40.879 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:43:40.951 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:40.950+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:40.951 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:40.950+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:40.951 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:40.950+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:40.951 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:40.950+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:40.951 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:40.950+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:41.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:41.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:41.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:41.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:41.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:41.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:41.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:41.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:41.388 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:41.387+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3323 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:41.397 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:41.396+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:41.397 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:41.396+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:41.397 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:41.396+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:41.397 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:41.396+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:41.656 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:41.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:41.656 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:41.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:41.656 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:41.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:41.656 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:41.655+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:41.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:41.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:41.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:41.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:41.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:41.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:41.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:41.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:41.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:41.906+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:42.322 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:42.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:42.322 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:42.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:42.322 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:42.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:42.322 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:42.321+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:42.351 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:42.351+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:42.351 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:42.351+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:42.352 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:42.351+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:42.352 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:42.351+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:42.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:42.684+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:42.684 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:42.684+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:42.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:42.684+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:42.685 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:42.684+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:42.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:42.927+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:42.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:42.927+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:42.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:42.927+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:42.928 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:42.927+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:42.929 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:42.927+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:43.281 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:43.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:43.281 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:43.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:43.282 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:43.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:43.282 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:43.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:43.322 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:43.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:43.322 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:43.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:43.322 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:43.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:43.322 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:43.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:43.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:43.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:43.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:43.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:43.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:43.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:43.706 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:43.706+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:43.935 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:43.934+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:43.935 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:43.934+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:43.935 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:43.934+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:43.935 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:43.934+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:43.935 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:43.934+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:44.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:44.320+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:44.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:44.320+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:44.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:44.320+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:44.321 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:44.320+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:44.343 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:44.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:44.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:44.342+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:44.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:44.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:44.344 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:44.343+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:44.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~215s 2023-07-04T23:43:44.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2023-07-04T23:43:44.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~215s 2023-07-04T23:43:44.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:44.688+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:44.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:44.688+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:44.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:44.688+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:44.688 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:44.688+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:44.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:44.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:44.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:44.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:44.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:44.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:44.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:44.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:44.956 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:44.955+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:45.316 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:45.316+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:45.316 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:45.316+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:45.317 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:45.316+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:45.317 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:45.316+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:45.336 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:45.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:45.336 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:45.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:45.337 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:45.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:45.337 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:45.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:45.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:45.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:45.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:45.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:45.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:45.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:45.708 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:45.708+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:45.923 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:45.923+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:45.924 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:45.923+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:45.924 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:45.923+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:45.924 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:45.923+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:45.924 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:45.923+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:46.279 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:46.279+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:46.279 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:46.279+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:46.280 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:46.279+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:46.280 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:46.279+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:46.356 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:46.356+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:46.356 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:46.356+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:46.356 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:46.356+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:46.356 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:46.356+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:46.389 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:46.388+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3400 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:46.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:46.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:46.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:46.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:46.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:46.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:46.712 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:46.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:46.971 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:46.970+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:46.971 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:46.970+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:46.972 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:46.970+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:46.972 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:46.970+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:46.972 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:46.970+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:47.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:47.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:47.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:47.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:47.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:47.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:47.261 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:47.260+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:47.340 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:47.339+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:47.340 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:47.339+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:47.340 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:47.339+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:47.340 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:47.339+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:47.726 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:47.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:47.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:47.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:47.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:47.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:47.727 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:47.725+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:47.954 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:47.954+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:47.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:47.954+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:47.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:47.954+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:47.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:47.954+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:47.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:47.954+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:48.294 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:48.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:48.294 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:48.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:48.295 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:48.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:48.295 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:48.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:48.321 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:48.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:48.321 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:48.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:48.321 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:48.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:48.322 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:48.321+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:48.736 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:48.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:48.736 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:48.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:48.736 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:48.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:48.736 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:48.735+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:48.915 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:48.914+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:48.915 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:48.914+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:48.916 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:48.914+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:48.916 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:48.914+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:48.916 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:48.914+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:49.314 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:49.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:49.314 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:49.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:49.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:49.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:49.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:49.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:49.325 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:49.325+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:49.325 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:49.325+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:49.325 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:49.325+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:49.325 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:49.325+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:49.704 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:49.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:49.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:49.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:49.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:49.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:49.705 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:49.704+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:49.891 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:49.891+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:49.891 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:49.891+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:49.892 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:49.891+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:49.892 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:49.891+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:49.892 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:49.891+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:50.310 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:50.310+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:50.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:50.310+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:50.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:50.310+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:50.311 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:50.310+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:50.375 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:50.375+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:50.375 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:50.375+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:50.376 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:50.375+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:50.376 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:50.375+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:50.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~221s 2023-07-04T23:43:50.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~221s 2023-07-04T23:43:50.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~221s 2023-07-04T23:43:50.675 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:50.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:50.675 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:50.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:50.675 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:50.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:50.676 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:50.675+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:50.875 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:50.875+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:50.875 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:50.875+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:50.875 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:50.875+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:50.875 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:50.875+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:50.876 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:50.875+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:51.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:51.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:51.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:51.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:51.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:51.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:51.285 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:51.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:51.374 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:51.373+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:51.374 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:51.373+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:51.374 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:51.373+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:51.374 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:51.373+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:51.390 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:51.389+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3487 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:51.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:51.712+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:51.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:51.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:51.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:51.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:51.714 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:51.713+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:51.906 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:51.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:51.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:51.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:51.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:51.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:51.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:51.906+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:51.907 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:51.906+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:52.252 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:52.252+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:52.252 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:52.252+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:52.252 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:52.252+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:52.252 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:52.252+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:52.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:52.366+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:52.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:52.366+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:52.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:52.366+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:52.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:52.366+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:52.760 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:52.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:52.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:52.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:52.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:52.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:52.761 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:52.760+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:52.864 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:52.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:52.864 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:52.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:52.864 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:52.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:52.864 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:52.863+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:52.864 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:52.863+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:53.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:53.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:53.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:53.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:53.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:53.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:53.263 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:53.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:53.411 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:53.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:53.411 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:53.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:53.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:53.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:53.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:53.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:53.800 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:53.800+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:53.801 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:53.800+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:53.801 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:53.800+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:53.801 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:53.800+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:53.880 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:53.880+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:53.881 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:53.880+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:53.881 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:53.880+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:53.881 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:53.880+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:53.881 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:53.880+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:54.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:54.288+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:54.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:54.288+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:54.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:54.288+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:54.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:54.288+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:54.395 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:54.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:54.395 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:54.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:54.395 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:54.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:54.396 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:54.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:54.795 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:54.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:54.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:54.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:54.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:54.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:54.796 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:54.795+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:54.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:54.885+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:54.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:54.885+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:54.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:54.885+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:54.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:54.885+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:54.885 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:54.885+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:55.274 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:55.273+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:55.274 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:55.273+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:55.274 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:55.273+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:55.274 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:55.273+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:55.346 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:55.346+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:55.346 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:55.346+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:55.346 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:55.346+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:55.346 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:55.346+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:55.837 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:55.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:55.838 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:55.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:55.838 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:55.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:55.838 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:55.837+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:55.870 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:55.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:55.870 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:55.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:55.870 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:55.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:55.871 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:55.870+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:55.871 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:55.870+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:56.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~227s 2023-07-04T23:43:56.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~227s 2023-07-04T23:43:56.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~227s 2023-07-04T23:43:56.290 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:56.290+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:56.290 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:56.290+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:56.290 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:56.290+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:56.291 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:56.290+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:56.301 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:56.302+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:56.301 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:56.302+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:56.302 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:56.302+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:56.302 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:56.302+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:56.390 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:43:56.391+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3564 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:43:56.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:56.858+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:56.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:56.858+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:56.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:56.858+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:56.858 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:56.858+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:56.920 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:56.920+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:56.920 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:56.920+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:56.920 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:56.920+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:56.920 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:56.920+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:56.921 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:56.920+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:57.302 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:57.302+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:57.302 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:57.302+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:57.302 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:57.302+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:57.302 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:57.302+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:57.340 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:57.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:57.340 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:57.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:57.340 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:57.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:57.340 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:57.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:57.845 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:57.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:57.845 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:57.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:57.845 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:57.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:57.845 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:57.845+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:57.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:57.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:57.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:57.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:57.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:57.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:57.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:57.955+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:57.955 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:57.955+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:58.305 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:58.306+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:58.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:58.306+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:58.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:58.306+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:58.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:58.306+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:58.359 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:58.359+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:58.359 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:58.359+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:58.359 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:58.359+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:58.360 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:58.359+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:58.868 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:58.867+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:58.868 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:58.867+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:58.869 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:58.867+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:58.869 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:58.867+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:58.943 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:58.943+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:58.944 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:58.943+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:58.944 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:58.943+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:58.944 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:58.943+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:58.944 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:58.943+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:43:59.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:59.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:59.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:59.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:59.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:59.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:43:59.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:43:59.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:43:59.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:59.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:59.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:59.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:59.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:59.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:43:59.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:43:59.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:43:59.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:59.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:59.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:59.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:43:59.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:59.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:59.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:43:59.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:43:59.993 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:59.992+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:59.993 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:59.992+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:59.993 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:59.992+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:43:59.993 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:59.992+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:43:59.994 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:43:59.992+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:00.258 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:00.257+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:00.258 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:00.257+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:00.258 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:00.257+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:00.259 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:00.257+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:00.409 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:00.409+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:00.410 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:00.409+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:00.410 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:00.409+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:00.410 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:00.409+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:00.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:00.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:00.788 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:00.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:00.789 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:00.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:00.789 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:00.787+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:00.962 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:00.961+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:00.963 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:00.961+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:00.963 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:00.961+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:00.963 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:00.961+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:00.963 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:00.961+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:01.239 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:01.238+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:01.239 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:01.238+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:01.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:01.238+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:01.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:01.238+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:01.391 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:01.390+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3640 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:01.453 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:01.452+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:01.453 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:01.452+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:01.453 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:01.452+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:01.453 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:01.452+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:01.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:01.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:01.833 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:01.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:01.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:01.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:01.834 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:01.832+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:01.978 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:01.977+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:01.978 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:01.977+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:01.978 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:01.977+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:01.978 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:01.977+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:01.978 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:01.977+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:02.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~232s 2023-07-04T23:44:02.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~232s 2023-07-04T23:44:02.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~232s 2023-07-04T23:44:02.070 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~232s 2023-07-04T23:44:02.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~232s 2023-07-04T23:44:02.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~232s 2023-07-04T23:44:02.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~232s 2023-07-04T23:44:02.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~232s 2023-07-04T23:44:02.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~232s 2023-07-04T23:44:02.231 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:02.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:02.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:02.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:02.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:02.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:02.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:02.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:02.479 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:02.478+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:02.479 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:02.478+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:02.480 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:02.478+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:02.480 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:02.478+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:02.876 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:02.875+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:02.876 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:02.875+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:02.876 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:02.875+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:02.877 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:02.875+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:03.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:03.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.012+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:03.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:03.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:03.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:03.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:03.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:03.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:03.224 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:03.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:03.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:03.459+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:03.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:03.459+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:03.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:03.459+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:03.461 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:03.459+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:03.902 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:03.901+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:03.902 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:03.901+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:03.902 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:03.901+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:03.902 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:03.901+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:03.995 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.994+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.995 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.994+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.995 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.994+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:03.995 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.994+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:03.995 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:03.994+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:04.175 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:04.175 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:04.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:04.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:04.425 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:04.425+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:04.426 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:04.425+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:04.426 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:04.425+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:04.426 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:04.425+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:04.909 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:04.908+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:04.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:04.908+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:04.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:04.908+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:04.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:04.908+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:04.970 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:04.969+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:04.970 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:04.969+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:04.971 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:04.969+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:04.971 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:04.969+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:04.971 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:04.969+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:05.159 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:05.158+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:05.159 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:05.158+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:05.159 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:05.158+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:05.159 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:05.158+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:05.383 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:05.382+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:05.383 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:05.382+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:05.383 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:05.382+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:05.384 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:05.382+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:05.911 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:05.910+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:05.911 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:05.910+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:05.911 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:05.910+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:05.911 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:05.910+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:06.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:06.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:06.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:06.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:06.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:06.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:06.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:06.012+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:06.013 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:06.012+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:06.127 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:06.126+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:06.127 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:06.126+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:06.127 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:06.126+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:06.128 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:06.126+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:06.392 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:06.391+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3727 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:06.420 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:06.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:06.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:06.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:06.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:06.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:06.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:06.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:06.903 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:06.902+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:06.903 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:06.902+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:06.904 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:06.902+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:06.904 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:06.902+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:07.049 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:07.049+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:07.050 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:07.049+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:07.050 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:07.049+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:07.050 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:07.049+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:07.050 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:07.049+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:07.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:07.102+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:07.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:07.102+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:07.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:07.102+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:07.104 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:07.102+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:07.441 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:07.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:07.441 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:07.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:07.442 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:07.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:07.442 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:07.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:07.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~238s 2023-07-04T23:44:07.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~238s 2023-07-04T23:44:07.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~238s 2023-07-04T23:44:07.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~238s 2023-07-04T23:44:07.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~238s 2023-07-04T23:44:07.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~238s 2023-07-04T23:44:07.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~238s 2023-07-04T23:44:07.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~238s 2023-07-04T23:44:07.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~238s 2023-07-04T23:44:07.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:07.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:07.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:07.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:07.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:07.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:07.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:07.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:08.055 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:08.054+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:08.055 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:08.054+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:08.056 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:08.054+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:08.056 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:08.054+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:08.056 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:08.054+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:08.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:08.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:08.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:08.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:08.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:08.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:08.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:08.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:08.445 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:08.444+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:08.446 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:08.445+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:08.446 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:08.445+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:08.446 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:08.445+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:08.914 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:08.913+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:08.914 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:08.913+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:08.914 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:08.913+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:08.915 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:08.913+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:09.104 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:09.103+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:09.104 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:09.103+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:09.104 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:09.103+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:09.105 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:09.103+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:09.105 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:09.103+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:09.124 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:09.123+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:09.124 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:09.123+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:09.124 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:09.123+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:09.124 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:09.123+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:09.471 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:09.471+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:09.472 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:09.471+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:09.472 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:09.471+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:09.472 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:09.471+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:09.905 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:09.904+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:09.906 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:09.904+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:09.906 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:09.904+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:09.906 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:09.904+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:10.142 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:10.141+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:10.143 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:10.141+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:10.143 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:10.141+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:10.143 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:10.141+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:10.143 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:10.142+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:10.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:10.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:10.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:10.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:10.153 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:10.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:10.153 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:10.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:10.433 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:10.432+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:10.433 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:10.432+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:10.433 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:10.432+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:10.433 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:10.432+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:10.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:44:10.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:44:10.955 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:10.954+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:10.956 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:10.954+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:10.956 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:10.954+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:10.956 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:10.954+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:11.129 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:11.128+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:11.129 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:11.128+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:11.129 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:11.128+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:11.129 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:11.128+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:11.130 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:11.128+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:11.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:11.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:11.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:11.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:11.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:11.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:11.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:11.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:11.393 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:11.392+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3805 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:11.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:11.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:11.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:11.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:11.424 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:11.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:11.424 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:11.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:11.938 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:11.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:11.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:11.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:11.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:11.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:11.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:11.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:12.165 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:12.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:12.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:12.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:12.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:12.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:12.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:12.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:12.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:12.165+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:12.188 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:12.187+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:12.188 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:12.187+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:12.188 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:12.187+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:12.188 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:12.187+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:12.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:12.420+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:12.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:12.420+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:12.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:12.420+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:12.421 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:12.420+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:12.938 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:12.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:12.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:12.939 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:12.938+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:13.137 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:13.137+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:13.138 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:13.137+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:13.138 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:13.137+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:13.138 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:13.137+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:13.138 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:13.137+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:13.237 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:13.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:13.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:13.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:13.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:13.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:13.238 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:13.237+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:13.394 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:13.393+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:13.394 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:13.393+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:13.394 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:13.393+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:13.394 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:13.393+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~244s 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~244s 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~244s 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~244s 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~244s 2023-07-04T23:44:13.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~244s 2023-07-04T23:44:13.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~244s 2023-07-04T23:44:13.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~244s 2023-07-04T23:44:13.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~244s 2023-07-04T23:44:13.919 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:13.918+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:13.919 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:13.918+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:13.920 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:13.918+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:13.920 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:13.918+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:14.096 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:14.095+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:14.096 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:14.095+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:14.096 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:14.095+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:14.096 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:14.095+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:14.096 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:14.095+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:14.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:14.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:14.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:14.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:14.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:14.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:14.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:14.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:14.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:14.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:14.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:14.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:14.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:14.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:14.397 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:14.396+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:14.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:14.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:14.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:14.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:14.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:14.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:14.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:14.943+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:15.100 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:15.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:15.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:15.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:15.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:15.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:15.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:15.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:15.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:15.100+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:15.241 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:15.241+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:15.242 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:15.241+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:15.242 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:15.241+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:15.242 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:15.241+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:15.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:15.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:15.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:15.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:15.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:15.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:15.412 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:15.411+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:15.970 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:15.969+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:15.970 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:15.969+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:15.970 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:15.969+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:15.970 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:15.969+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:16.089 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:16.088+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:16.089 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:16.088+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:16.089 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:16.088+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:16.089 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:16.088+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:16.089 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:16.088+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:16.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:16.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:16.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:16.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:16.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:16.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:16.263 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:16.262+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:16.394 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:16.393+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3897 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:16.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:16.460+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:16.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:16.460+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:16.460 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:16.460+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:16.461 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:16.460+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:16.989 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:16.988+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:16.989 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:16.988+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:16.989 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:16.988+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:16.989 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:16.988+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:17.051 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:17.050+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:17.051 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:17.050+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:17.051 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:17.050+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:17.051 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:17.050+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:17.052 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:17.050+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:17.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:17.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:17.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:17.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:17.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:17.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:17.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:17.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:17.447 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:17.447+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:17.447 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:17.447+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:17.448 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:17.447+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:17.448 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:17.447+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:17.944 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:17.944+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:17.945 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:17.944+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:17.945 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:17.944+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:17.945 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:17.944+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:18.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:18.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:18.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:18.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:18.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:18.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:18.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:18.046+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:18.047 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:18.046+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:18.209 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:18.208+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:18.209 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:18.208+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:18.209 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:18.208+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:18.209 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:18.208+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:18.442 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:18.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:18.443 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:18.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:18.443 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:18.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:18.443 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:18.441+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:18.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:18.909+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:18.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:18.909+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:18.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:18.909+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:18.910 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:18.909+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:19.091 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:19.090+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:19.091 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:19.090+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:19.091 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:19.090+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:19.091 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:19.090+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:19.091 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:19.090+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:19.255 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:19.254+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:19.255 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:19.254+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:19.255 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:19.254+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:19.255 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:19.254+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:19.472 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:19.472+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:19.472 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:19.472+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:19.473 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:19.472+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:19.473 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:19.472+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:19.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~250s 2023-07-04T23:44:19.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~250s 2023-07-04T23:44:19.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~250s 2023-07-04T23:44:19.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:19.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:19.949 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:19.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:19.950 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:19.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:19.950 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:19.948+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:20.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:20.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:20.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:20.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:20.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:20.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:20.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:20.113+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:20.114 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:20.113+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:20.206 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:20.205+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:20.206 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:20.205+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:20.206 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:20.205+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:20.206 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:20.205+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:20.485 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:20.485+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:20.486 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:20.485+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:20.486 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:20.485+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:20.486 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:20.485+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:20.972 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:20.972+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:20.972 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:20.972+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:20.973 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:20.972+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:20.973 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:20.972+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:21.100 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:21.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:21.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:21.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:21.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:21.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:21.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:21.100+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:21.101 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:21.100+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:21.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:21.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:21.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:21.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:21.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:21.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:21.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:21.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:21.395 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:21.394+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 3985 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:21.499 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:21.497+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:21.499 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:21.497+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:21.499 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:21.497+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:21.499 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:21.497+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:21.975 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:21.975+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:21.976 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:21.975+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:21.976 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:21.975+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:21.976 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:21.975+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:22.122 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:22.122+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:22.123 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:22.122+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:22.123 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:22.122+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:22.123 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:22.122+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:22.123 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:22.122+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:22.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:22.215+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:22.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:22.215+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:22.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:22.215+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:22.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:22.215+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:22.500 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:22.500+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:22.500 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:22.500+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:22.500 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:22.500+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:22.501 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:22.500+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:22.983 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:22.982+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:23.135 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:23.135+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:23.135 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:23.135+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:23.136 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:23.135+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:23.136 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:23.135+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:23.136 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:23.135+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:23.234 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:23.234+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:23.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:23.234+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:23.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:23.234+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:23.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:23.234+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:23.549 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:23.548+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:23.549 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:23.548+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:23.549 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:23.548+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:23.549 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:23.548+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:24.007 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:24.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:24.007 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:24.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:24.007 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:24.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:24.008 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:24.007+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:24.165 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:24.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:24.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:24.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:24.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:24.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:24.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:24.164+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:24.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:24.164+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:24.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:24.223+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:24.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:24.223+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:24.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:24.223+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:24.224 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:24.223+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:24.580 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:24.580+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:24.580 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:24.580+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:24.580 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:24.580+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:24.580 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:24.580+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:25.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:25.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:25.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:25.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:25.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:25.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:25.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:25.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:25.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:25.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:25.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:25.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:25.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:25.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:25.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:25.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:25.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:25.173+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:25.267 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:25.266+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:25.267 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:25.266+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:25.267 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:25.266+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:25.267 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:25.266+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~256s 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~256s 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~256s 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~256s 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~256s 2023-07-04T23:44:25.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~256s 2023-07-04T23:44:25.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~256s 2023-07-04T23:44:25.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~256s 2023-07-04T23:44:25.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~256s 2023-07-04T23:44:25.563 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:25.563+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:25.563 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:25.563+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:25.563 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:25.563+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:25.564 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:25.563+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:26.050 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:26.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:26.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:26.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:26.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:26.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:26.051 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:26.050+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:26.145 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:26.144+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:26.145 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:26.144+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:26.145 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:26.144+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:26.145 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:26.144+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:26.146 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:26.144+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:26.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:26.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:26.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:26.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:26.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:26.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:26.315 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:26.314+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:26.396 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:26.395+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4064 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:26.529 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:26.529+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:26.529 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:26.529+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:26.529 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:26.529+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:26.529 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:26.529+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:27.069 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:27.068+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:27.069 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:27.068+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:27.069 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:27.068+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:27.069 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:27.068+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:27.134 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:27.133+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:27.134 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:27.133+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:27.134 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:27.133+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:27.134 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:27.133+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:27.134 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:27.133+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:27.274 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:27.274+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:27.275 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:27.274+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:27.275 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:27.274+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:27.275 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:27.274+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:27.521 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:27.521+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:27.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:27.521+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:27.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:27.521+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:27.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:27.521+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:28.108 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:28.107+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:28.108 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:28.107+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:28.108 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:28.107+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:28.108 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:28.107+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:28.127 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:28.127+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:28.127 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:28.127+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:28.127 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:28.127+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:28.127 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:28.127+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:28.128 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:28.127+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:28.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:28.240+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:28.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:28.240+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:28.241 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:28.240+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:28.241 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:28.240+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:28.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:28.522+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:28.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:28.522+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:28.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:28.522+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:28.522 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:28.522+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:29.113 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:29.112+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:29.113 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:29.112+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:29.113 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:29.112+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:29.113 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:29.112+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:29.165 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:29.165+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:29.165 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:29.165+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:29.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:29.165+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:29.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:29.165+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:29.166 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:29.165+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:29.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:29.216+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:29.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:29.216+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:29.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:29.216+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:29.216 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:29.216+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:29.512 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:29.512+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:29.513 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:29.512+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:29.513 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:29.512+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:29.513 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:29.512+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:30.119 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:30.119+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:30.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:30.119+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:30.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:30.119+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:30.120 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:30.119+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:30.195 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:30.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:30.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:30.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:30.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:30.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:30.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:30.195+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:30.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:30.195+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:30.228 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:30.228+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:30.228 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:30.228+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:30.228 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:30.228+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:30.228 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:30.228+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:30.511 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:30.511+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:30.512 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:30.511+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:30.512 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:30.511+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:30.512 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:30.511+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:31.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~261s 2023-07-04T23:44:31.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~261s 2023-07-04T23:44:31.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~261s 2023-07-04T23:44:31.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~261s 2023-07-04T23:44:31.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~261s 2023-07-04T23:44:31.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~261s 2023-07-04T23:44:31.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~261s 2023-07-04T23:44:31.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~261s 2023-07-04T23:44:31.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~261s 2023-07-04T23:44:31.127 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:31.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:31.127 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:31.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:31.127 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:31.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:31.128 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:31.127+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:31.214 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:31.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:31.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:31.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:31.215 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:31.213+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:31.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:31.227+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:31.227 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:31.227+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:31.228 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:31.227+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:31.228 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:31.227+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:31.228 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:31.227+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:31.397 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:31.396+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4146 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:31.464 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:31.464+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:31.464 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:31.464+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:31.464 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:31.464+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:31.464 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:31.464+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:32.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:32.154+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:32.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:32.154+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:32.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:32.154+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:32.155 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:32.154+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:32.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:32.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:32.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:32.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:32.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:32.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:32.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:32.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:32.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:32.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:32.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:32.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:32.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:32.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:32.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:32.267+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:32.267 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:32.267+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:32.420 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:32.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:32.420 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:32.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:32.420 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:32.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:32.420 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:32.419+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:33.134 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:33.134+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:33.134 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:33.134+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:33.134 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:33.134+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:33.135 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:33.134+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:33.230+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:33.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:33.230+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:33.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:33.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:33.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:33.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:33.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:33.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:33.233 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:33.232+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:33.396 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:33.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:33.396 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:33.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:33.396 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:33.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:33.396 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:33.395+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:34.175 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:34.175+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:34.176 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:34.175+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:34.176 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:34.175+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:34.176 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:34.175+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:34.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:34.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:34.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:34.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:34.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:34.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:34.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:34.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:34.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:34.276+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:34.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:34.276+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:34.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:34.276+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:34.276 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:34.276+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:34.277 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:34.276+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:34.349 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:34.349+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:34.349 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:34.349+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:34.349 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:34.349+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:34.349 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:34.349+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:35.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:35.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:35.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:35.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:35.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:35.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:35.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:35.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:35.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:35.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:35.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:35.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:35.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:35.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:35.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:35.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:35.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:35.231+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:35.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:35.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:35.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:35.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:35.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:35.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:35.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:35.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:35.328 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:35.327+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:35.328 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:35.327+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:35.328 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:35.327+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:35.328 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:35.327+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:36.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:36.216+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:36.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:36.216+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:36.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:36.216+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:36.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:36.216+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:36.218 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:36.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:36.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:36.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:36.218+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:36.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:36.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:36.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:36.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:36.231 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:36.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:36.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:36.231+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:36.232 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:36.231+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:36.289 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:36.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:36.290 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:36.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:36.290 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:36.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:36.290 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:36.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:36.398 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:36.397+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4231 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:36.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~267s 2023-07-04T23:44:36.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~267s 2023-07-04T23:44:36.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~267s 2023-07-04T23:44:36.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~267s 2023-07-04T23:44:36.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~267s 2023-07-04T23:44:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~267s 2023-07-04T23:44:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~267s 2023-07-04T23:44:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~267s 2023-07-04T23:44:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~267s 2023-07-04T23:44:37.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:37.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:37.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:37.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:37.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:37.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:37.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:37.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:37.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:37.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:37.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:37.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:37.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:37.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:37.199 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:37.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:37.252 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:37.252 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:37.252 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:37.252 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:37.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:37.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:37.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:37.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:37.253 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:37.251+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:38.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:38.154+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:38.155 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:38.154+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:38.155 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:38.154+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:38.155 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:38.154+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:38.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:38.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:38.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:38.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:38.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:38.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:38.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:38.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:38.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:38.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:38.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:38.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:38.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:38.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:38.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:38.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:38.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:38.240+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:38.281 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:38.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:38.282 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:38.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:38.282 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:38.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:38.282 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:38.281+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:39.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:39.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:39.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:39.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:39.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:39.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:39.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:39.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:39.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:39.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:39.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:39.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:39.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:39.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:39.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:39.192+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:39.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:39.192+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:39.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:39.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:39.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:39.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:39.196 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:39.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:39.197 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:39.195+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:39.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:39.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:39.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:39.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:39.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:39.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:39.264 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:39.263+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:40.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:40.186+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:40.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:40.186+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:40.187 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:40.186+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:40.187 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:40.186+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:40.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:40.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:40.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:40.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:40.196 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:40.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:40.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:40.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:40.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:40.196+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:40.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:40.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:40.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:40.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:40.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:40.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:40.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:40.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:40.289 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:40.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:40.289 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:40.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:40.290 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:40.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:40.290 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:40.289+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:41.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:44:41.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:44:41.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:41.198+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:41.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:41.198+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:41.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:41.198+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:41.198 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:41.198+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:41.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:41.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:41.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:41.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:41.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:41.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:41.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:41.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:41.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:41.216+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:41.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:41.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:41.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:41.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:41.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:41.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:41.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:41.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:41.300 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:41.300+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:41.301 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:41.300+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:41.301 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:41.300+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:41.301 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:41.300+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:41.400 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:41.399+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4309 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:42.165 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:42.165+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:42.165 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:42.165+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:42.166 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:42.165+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:42.166 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:42.165+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:42.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:42.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:42.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:42.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:42.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:42.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:42.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:42.242+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:42.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:42.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:42.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:42.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:42.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:42.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:42.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:42.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:42.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:42.264+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:42.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:42.345+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:42.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:42.345+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:42.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:42.345+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:42.346 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:42.345+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:42.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~273s 2023-07-04T23:44:42.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~273s 2023-07-04T23:44:42.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~273s 2023-07-04T23:44:43.184 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:43.184+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:43.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:43.184+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:43.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:43.184+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:43.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:43.184+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:43.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:43.217+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:43.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:43.217+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:43.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:43.217+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:43.217 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:43.217+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:43.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:43.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:43.310 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:43.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:43.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:43.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:43.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:43.310+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:43.311 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:43.310+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:43.328 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:43.329+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:43.329 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:43.329+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:43.329 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:43.329+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:43.329 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:43.329+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:44.169 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:44.168+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:44.169 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:44.168+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:44.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:44.168+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:44.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:44.168+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:44.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:44.222+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:44.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:44.222+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:44.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:44.222+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:44.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:44.222+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:44.274 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:44.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:44.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:44.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:44.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:44.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:44.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:44.273+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:44.275 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:44.273+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:44.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:44.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:44.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:44.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:44.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:44.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:44.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:44.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:45.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:45.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:45.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:45.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:45.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:45.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:45.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:45.138+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:45.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:45.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:45.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:45.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:45.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:45.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:45.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:45.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:45.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:45.237+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:45.263 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:45.262+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:45.263 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:45.262+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:45.263 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:45.262+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:45.263 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:45.262+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:45.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:45.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:45.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:45.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:45.341 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:45.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:45.342 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:45.340+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:46.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:46.141+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:46.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:46.141+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:46.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:46.141+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:46.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:46.141+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:46.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:46.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:46.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:46.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:46.219 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:46.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:46.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:46.218+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:46.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:46.218+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:46.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:46.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:46.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:46.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:46.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:46.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:46.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:46.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:46.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:46.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:46.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:46.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:46.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:46.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:46.345 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:46.344+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:46.400 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:46.399+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4386 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:47.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:47.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:47.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:47.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:47.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:47.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:47.169 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:47.167+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:47.172 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:47.171+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:47.172 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:47.171+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:47.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:47.171+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:47.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:47.171+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:47.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:47.171+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:47.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:47.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:47.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:47.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:47.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:47.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:47.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:47.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:47.385 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:47.384+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:47.385 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:47.384+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:47.385 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:47.384+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:47.385 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:47.384+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:48.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:48.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:48.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:48.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:48.194 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:48.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:48.194 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:48.192+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:48.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:48.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:48.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:48.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:48.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:48.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:48.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:48.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:48.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:48.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:48.266 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:48.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:48.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:48.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:48.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:48.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:48.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:48.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:48.379 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:48.378+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:48.379 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:48.378+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:48.380 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:48.378+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:48.380 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:48.378+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:48.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~279s 2023-07-04T23:44:48.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~279s 2023-07-04T23:44:48.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~279s 2023-07-04T23:44:48.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~279s 2023-07-04T23:44:49.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:49.191+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:49.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:49.191+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:49.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:49.191+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:49.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:49.191+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:49.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:49.191+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:49.210 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:49.210+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:49.211 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:49.210+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:49.211 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:49.210+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:49.211 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:49.210+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:49.259 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:49.258+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:49.259 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:49.258+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:49.259 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:49.258+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:49.259 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:49.258+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:49.422 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:49.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:49.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:49.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:49.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:49.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:49.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:49.422+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:50.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:50.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:50.177 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:50.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:50.177 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:50.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:50.177 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:50.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:50.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:50.235+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:50.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:50.235+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:50.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:50.235+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:50.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:50.235+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:50.236 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:50.235+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:50.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:50.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:50.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:50.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:50.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:50.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:50.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:50.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:50.468 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:50.467+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:50.469 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:50.467+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:50.469 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:50.468+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:50.469 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:50.468+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:51.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:51.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:51.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:51.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:51.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:51.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:51.152 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:51.151+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:51.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:51.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:51.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:51.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:51.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:51.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:51.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:51.209+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:51.210 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:51.209+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:51.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:51.296+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:51.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:51.296+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:51.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:51.296+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:51.298 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:51.296+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:51.402 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:51.401+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4472 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:51.471 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:51.470+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:51.471 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:51.470+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:51.471 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:51.470+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:51.471 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:51.470+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:52.160 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:52.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:52.161 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:52.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:52.161 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:52.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:52.161 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:52.159+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:52.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:52.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:52.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:52.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:52.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:52.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:52.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:52.196+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:52.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:52.196+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:52.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:52.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:52.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:52.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:52.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:52.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:52.256 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:52.255+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:52.422 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:52.421+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:52.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:52.421+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:52.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:52.421+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:52.423 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:52.421+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:53.114 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:53.113+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:53.114 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:53.113+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:53.115 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:53.113+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:53.115 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:53.113+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:53.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:53.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:53.237 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:53.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:53.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:53.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:53.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:53.236+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:53.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:53.236+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:53.304 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:53.303+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:53.304 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:53.303+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:53.304 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:53.303+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:53.304 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:53.303+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:53.392 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:53.391+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:53.392 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:53.391+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:53.392 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:53.391+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:53.392 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:53.391+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:54.138 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:54.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:54.138 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:54.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:54.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:54.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:54.139 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:54.137+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:54.229 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:54.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:54.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:54.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:54.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:54.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:54.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:54.228+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:54.230 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:54.228+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~285s 2023-07-04T23:44:54.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~285s 2023-07-04T23:44:54.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~285s 2023-07-04T23:44:54.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~285s 2023-07-04T23:44:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:54.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:54.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:54.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:54.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:54.370 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:54.369+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:54.370 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:54.369+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:54.371 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:54.369+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:54.371 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:54.369+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:55.167 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:55.166+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:55.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:55.166+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:55.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:55.166+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:55.168 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:55.166+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:55.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:55.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:55.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:55.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:55.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:55.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:55.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:55.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:55.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:55.263+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:55.334 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:55.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:55.334 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:55.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:55.334 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:55.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:55.334 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:55.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:55.375 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:55.374+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:55.376 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:55.374+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:55.376 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:55.374+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:55.376 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:55.374+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:56.178 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:56.178+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:56.179 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:56.178+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:56.179 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:56.178+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:56.179 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:56.178+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:56.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:56.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:56.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:56.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:56.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:56.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:56.264 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:56.263+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:56.265 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:56.263+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:56.330 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:56.329+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:56.330 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:56.329+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:56.330 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:56.329+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:56.330 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:56.329+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:56.377 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:56.377+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:56.378 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:56.377+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:56.378 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:56.377+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:56.378 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:56.377+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:56.402 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:44:56.402+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4549 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:44:57.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:57.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:57.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:57.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:57.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:57.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:57.183 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:57.182+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:57.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:57.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:57.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:57.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:57.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:57.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:57.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:57.214+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:57.215 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:57.214+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:57.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:57.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:57.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:57.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:57.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:57.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:57.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:57.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:57.365 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:57.365+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:57.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:57.365+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:57.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:57.365+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:57.366 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:57.365+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:58.175 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:58.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:58.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:58.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:58.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:58.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:58.176 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:58.175+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:58.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:58.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:58.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:58.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:58.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:58.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:58.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:58.242+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:58.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:58.242+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:58.306 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:58.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:58.306 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:58.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:58.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:58.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:58.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:58.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:58.405 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:58.404+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:58.405 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:58.404+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:58.405 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:58.404+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:58.405 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:58.404+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:59.192 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:59.191+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:59.192 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:59.191+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:59.192 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:59.191+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:44:59.192 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:44:59.191+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:44:59.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:59.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:59.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:59.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:59.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:59.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:44:59.212 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:59.211+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:44:59.213 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:44:59.211+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:44:59.338 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:59.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:59.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:59.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:44:59.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:59.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:59.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:44:59.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:44:59.401 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:59.401+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:59.401 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:59.401+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:44:59.402 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:59.401+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:44:59.402 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:44:59.401+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:00.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~290s 2023-07-04T23:45:00.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~290s 2023-07-04T23:45:00.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~290s 2023-07-04T23:45:00.153 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:00.152+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:00.153 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:00.152+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:00.153 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:00.152+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:00.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:00.152+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:00.187 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:00.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:00.187 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:00.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:00.187 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:00.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:00.187 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:00.187+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:00.188 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:00.187+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:00.327 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:00.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:00.328 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:00.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:00.328 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:00.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:00.328 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:00.327+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:00.368 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:00.367+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:00.368 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:00.367+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:00.368 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:00.367+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:00.368 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:00.367+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:01.169 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:01.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:01.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:01.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:01.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:01.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:01.170 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:01.169+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:01.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:01.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:01.204 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:01.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:01.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:01.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:01.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:01.204+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:01.205 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:01.204+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:01.339 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:01.338+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:01.339 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:01.338+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:01.339 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:01.338+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:01.339 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:01.338+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:01.341 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:01.340+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:01.341 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:01.340+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:01.342 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:01.340+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:01.342 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:01.340+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:01.404 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:01.403+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4642 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:02.141 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:02.140+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:02.141 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:02.140+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:02.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:02.140+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:02.142 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:02.140+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:02.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:02.223+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:02.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:02.223+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:02.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:02.223+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:02.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:02.223+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:02.224 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:02.223+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:02.294 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:02.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:02.295 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:02.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:02.295 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:02.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:02.295 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:02.294+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:02.381 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:02.380+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:02.381 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:02.380+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:02.381 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:02.380+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:02.381 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:02.380+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:03.093 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:03.092+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:03.093 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:03.092+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:03.094 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:03.092+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:03.094 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:03.092+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:03.206 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:03.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:03.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:03.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:03.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:03.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:03.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:03.205+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:03.207 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:03.205+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:03.268 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:03.267+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:03.268 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:03.267+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:03.268 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:03.267+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:03.268 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:03.267+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:03.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:03.407+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:03.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:03.407+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:03.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:03.407+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:03.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:03.407+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:04.097 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:04.096+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:04.097 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:04.096+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:04.097 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:04.096+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:04.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:04.096+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:04.200 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:04.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:04.200 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:04.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:04.200 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:04.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:04.200 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:04.199+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:04.201 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:04.199+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:04.230 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:04.230+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:04.230 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:04.230+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:04.231 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:04.230+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:04.231 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:04.230+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:04.441 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:04.440+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:04.441 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:04.440+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:04.441 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:04.440+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:04.441 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:04.440+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:05.075 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:05.074+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:05.076 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:05.074+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:05.076 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:05.074+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:05.076 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:05.074+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:05.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:05.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:05.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:05.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:05.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:05.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:05.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:05.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:05.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:05.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:05.216 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:05.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:05.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:05.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:05.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:05.216+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:05.217 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:05.216+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:05.429 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:05.428+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:05.429 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:05.428+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:05.429 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:05.428+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:05.430 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:05.428+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:05.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~296s 2023-07-04T23:45:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~296s 2023-07-04T23:45:05.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~296s 2023-07-04T23:45:06.054 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:06.053+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:06.054 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:06.053+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:06.054 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:06.053+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:06.055 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:06.053+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:06.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:06.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:06.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:06.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:06.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:06.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:06.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:06.173+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:06.174 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:06.173+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:06.202 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:06.202+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:06.202 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:06.202+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:06.203 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:06.202+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:06.203 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:06.202+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:06.405 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:06.404+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4739 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:06.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:06.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:06.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:06.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:06.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:06.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:06.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:06.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:07.027 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.027+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.027 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.027+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.027 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.027+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.028 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.027+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:07.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:07.193+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:07.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:07.193+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:07.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:07.193+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:07.194 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:07.193+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:07.194 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:07.193+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:07.207 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:07.207+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:07.207 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:07.207+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:07.208 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:07.207+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:07.208 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:07.207+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:07.453 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:07.452+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:07.453 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:07.452+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:07.453 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:07.452+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:07.453 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:07.452+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:07.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:07.983 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:07.982+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:08.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:08.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:08.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:08.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:08.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:08.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:08.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:08.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:08.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:08.213+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:08.234 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:08.233+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:08.234 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:08.233+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:08.234 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:08.233+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:08.234 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:08.233+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:08.449 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:08.449+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:08.449 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:08.449+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:08.449 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:08.449+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:08.450 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:08.449+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:08.964 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:08.964+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:08.965 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:08.964+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:08.965 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:08.964+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:08.965 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:08.964+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:09.176 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:09.176+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:09.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:09.176+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:09.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:09.176+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:09.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:09.176+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:09.177 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:09.176+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:09.200 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:09.200+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:09.200 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:09.200+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:09.201 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:09.200+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:09.201 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:09.200+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:09.489 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:09.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:09.489 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:09.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:09.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:09.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:09.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:09.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:09.959 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:09.958+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:09.959 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:09.958+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:09.959 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:09.958+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:09.959 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:09.958+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:10.154 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:10.153+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:10.154 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:10.153+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:10.154 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:10.153+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:10.154 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:10.153+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:10.154 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:10.153+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:10.156 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:10.155+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:10.157 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:10.155+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:10.157 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:10.155+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:10.157 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:10.155+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:10.462 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:10.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:10.462 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:10.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:10.462 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:10.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:10.462 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:10.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:10.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:10.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:10.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:10.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:10.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:10.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:10.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:10.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:11.114 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:11.113+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:11.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:11.113+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:11.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:11.113+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:11.115 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:11.113+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:11.186 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:11.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:11.186 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:11.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:11.186 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:11.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:11.186 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:11.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:11.186 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:11.185+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:11.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:45:11.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:45:11.406 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:11.405+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4830 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:11.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:11.465+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:11.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:11.465+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:11.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:11.465+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:11.466 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:11.465+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:11.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~302s 2023-07-04T23:45:11.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~302s 2023-07-04T23:45:11.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~302s 2023-07-04T23:45:11.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~302s 2023-07-04T23:45:11.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~302s 2023-07-04T23:45:11.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~302s 2023-07-04T23:45:11.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~302s 2023-07-04T23:45:11.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~302s 2023-07-04T23:45:11.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~302s 2023-07-04T23:45:11.749 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-07-04T23:45:11.749 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T23:45:11.750 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T23:45:11.785 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T23:45:11.785 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T23:45:11.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-04T23:45:11.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:11.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:11.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:11.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:11.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:11.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:11.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:11.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:12.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:12.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:12.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:12.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:12.089 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:12.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:12.090 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:12.089+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:12.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:12.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:12.197 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:12.197+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:12.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:12.197+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:12.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.400+0000 7f71822fc700 1 Processor -- start 2023-07-04T23:45:12.401 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.400+0000 7f71822fc700 1 -- start start 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f71822fc700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f71822fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f71822fc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c0cb6d0 con 0x7f717c0aef10 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f71822fc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f717c0716a0 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f71822fc700 1 -- --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f717c06a550 2023-07-04T23:45:12.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f717b7fe700 1 -- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f717c0cbb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f717b7fe700 1 --2- >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f717bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f717c0aef10 0x7f717c069e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43180/0 (socket says 172.21.15.77:43180) 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36040/0 (socket says 172.21.15.77:36040) 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 learned_addr learned my addr 172.21.15.77:0/1941951922 (peer_addr_for_me v1:172.21.15.77:0/0) 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:12.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:12.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.401+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:12.442 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:12.441+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:12.442 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:12.441+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:12.442 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:12.441+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:12.442 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:12.441+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:12.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:12.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f717c0cbb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:12.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:12.603 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:12.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:12.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:12.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:12.602+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:12.986 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:12.985+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:12.986 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:12.985+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:12.986 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:12.985+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:12.986 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:12.985+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f717c0cbb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:13.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:13.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.003+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:13.107 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:13.107+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:13.107 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:13.107+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:13.107 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:13.107+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:13.107 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:13.107+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:13.243 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:13.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:13.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:13.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:13.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:13.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:13.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:13.243+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:13.244 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:13.243+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:13.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:13.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:13.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:13.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:13.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:13.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:13.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:13.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f717c0cbb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:13.804+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:14.035 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:14.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:14.035 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:14.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:14.035 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:14.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:14.035 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:14.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:14.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:14.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:14.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:14.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:14.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:14.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:14.136 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:14.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:14.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:14.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:14.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:14.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:14.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:14.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:14.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:14.213+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:14.214 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:14.213+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:14.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:14.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:14.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:14.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:14.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:14.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:14.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:14.489+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:15.007 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:15.006+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:15.008 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:15.006+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:15.008 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:15.006+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:15.008 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:15.006+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:15.124 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:15.124+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:15.125 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:15.124+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:15.125 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:15.124+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:15.125 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:15.124+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:15.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:15.220+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:15.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:15.220+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:15.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:15.220+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:15.220 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:15.220+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:15.221 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:15.220+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:15.402 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f717c0cbb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f717c0cbb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 msgr2=0x7f717c0cc0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f717c0716a0 0x7f717c0cc0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f717c0aef10 legacy=0x7f717c069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c0cb080 con 0x7f715c003770 2023-07-04T23:45:15.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0073f0 2023-07-04T23:45:15.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c003770 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:15.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:15.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.402+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:15.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:15.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:15.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:15.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:15.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:15.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:15.520 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:15.519+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:15.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.603+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c003770 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:15.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.603+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:15.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:15.603+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:16.004 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.004+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c003770 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:16.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.004+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:16.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.004+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:16.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.029+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.029+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.029+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.030 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.029+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:16.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:16.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:16.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:16.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:16.144 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:16.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:16.145 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:16.144+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:16.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:16.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:16.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:16.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:16.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:16.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:16.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:16.240+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:16.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:16.240+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:16.407 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:16.406+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4905 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:16.481 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:16.481+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:16.482 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:16.481+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:16.482 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:16.481+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:16.482 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:16.481+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:16.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.805+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c003770 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:16.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.805+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:16.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:16.805+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:16.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:16.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:16.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:17.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:17.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:17.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:17.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:17.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:17.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:17.184 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:17.183+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:17.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:17.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:17.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:17.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:17.198 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:17.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:17.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:17.198+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:17.199 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:17.198+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:17.484 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:17.484+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:17.485 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:17.484+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:17.485 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:17.484+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:17.485 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:17.484+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:17.993 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:17.993+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:17.994 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:17.993+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:17.994 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:17.993+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:17.994 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:17.993+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:18.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:18.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:18.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:18.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:18.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:18.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:18.135 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:18.135+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:18.163 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:18.163+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:18.163 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:18.163+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:18.164 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:18.163+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:18.164 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:18.163+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:18.164 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:18.163+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:18.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:18.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:18.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c003770 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:18.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:18.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00afb0 con 0x7f715c00a4c0 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0073f0 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f717c06a550 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f715c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:18.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00a4c0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:18.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:18.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.402+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:18.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:18.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:18.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:18.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:18.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:18.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:18.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:18.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:18.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.603+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00a4c0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:18.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.603+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:18.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.603+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f715c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:18.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.603+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:18.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:18.603+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:18.980 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:18.979+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:18.980 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:18.979+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:18.980 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:18.979+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:18.980 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:18.979+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.004+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f715c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.004+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.004+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00a4c0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.004+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.004+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:19.123 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:19.123+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:19.124 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:19.123+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:19.124 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:19.123+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:19.124 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:19.123+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:19.124 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:19.123+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:19.177 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:19.177+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:19.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:19.177+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:19.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:19.177+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:19.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:19.177+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:19.476 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:19.476+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:19.476 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:19.476+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:19.476 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:19.476+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:19.476 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:19.476+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.805+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.805+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00a4c0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.805+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f715c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.805+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:19.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:19.805+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:19.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:19.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:19.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:19.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:19.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:19.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:19.967 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:19.967+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:20.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:20.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:20.162 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:20.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:20.163 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:20.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:20.163 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:20.161+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:20.163 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:20.161+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:20.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:20.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:20.178 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:20.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:20.179 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:20.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:20.179 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:20.178+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:20.462 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:20.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:20.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:20.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:20.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:20.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:20.463 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:20.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:20.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:20.952+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:20.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:20.952+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:20.952 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:20.952+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:20.953 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:20.952+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:21.131 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:21.131+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:21.131 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:21.131+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:21.131 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:21.131+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:21.132 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:21.131+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:21.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:21.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:21.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:21.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:21.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:21.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:21.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:21.151+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:21.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:21.151+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:21.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 msgr2=0x7f715c009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:21.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f717c06a550 0x7f715c009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:21.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 msgr2=0x7f715c0098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0073f0 0x7f715c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00a4c0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717c0cb080 con 0x7f715c00d390 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c010e90 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:21.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.402+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:21.408 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:21.407+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 4991 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:21.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.407+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c00f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:45:21.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.407+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c00f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:45:21.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.407+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 0x7f715c00f770 :-1 s=OPENED pgs=3229 cs=1 l=1).handle_message read tag failed 2023-07-04T23:45:21.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.407+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 0x7f715c00f770 :-1 s=OPENED pgs=3229 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:45:21.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:21.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:21.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:21.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:21.493 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:21.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:21.494 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:21.493+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.603+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.603+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.603+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:21.604 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.603+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:21.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:21.603+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:21.907+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:21.907+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:21.907+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:21.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:21.907+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:22.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.004+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:22.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.004+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:22.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.004+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:22.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.004+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:22.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.004+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:22.134 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:22.134 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:22.134 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:22.134 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:22.134+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:22.150 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:22.150+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:22.150 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:22.150+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:22.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:22.150+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:22.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:22.150+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:22.151 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:22.150+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:22.473 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:22.473+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:22.474 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:22.473+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:22.474 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:22.473+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:22.474 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:22.473+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:22.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:22.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:22.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:22.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:22.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:22.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:22.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:22.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:22.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:22.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:22.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:22.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:22.861 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:22.860+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:23.153 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:23.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:23.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:23.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:23.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:23.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:23.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:23.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:23.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:23.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:23.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:23.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:23.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:23.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:23.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:23.168+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:23.169 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:23.168+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:23.488 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:23.488+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:23.489 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:23.488+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:23.489 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:23.488+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:23.489 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:23.488+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:23.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:23.877+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:23.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:23.877+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:23.877 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:23.877+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:23.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:23.877+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:24.185 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:24.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:24.185 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:24.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:24.185 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:24.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:24.185 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:24.185+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:24.185 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:24.185+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:24.196 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:24.197+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:24.197 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:24.197+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:24.197 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:24.197+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:24.197 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:24.197+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:24.403 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010e90 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c00f770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00afb0 con 0x7f715c00d390 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c009f60 2023-07-04T23:45:24.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013ef0 2023-07-04T23:45:24.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f60 legacy=0x7f715c010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:24.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:24.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.403+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:24.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:24.518+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:24.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:24.518+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:24.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:24.518+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:24.519 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:24.518+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:24.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.604+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:24.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.604+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f60 legacy=0x7f715c010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:24.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:24.604+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:24.865 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:24.865+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:24.865 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:24.865+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:24.865 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:24.865+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:24.866 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:24.865+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:25.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.005+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:25.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.005+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f60 legacy=0x7f715c010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:25.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.005+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:25.172 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:25.172+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:25.172 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:25.172+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:25.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:25.172+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:25.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:25.172+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:25.173 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:25.172+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:25.244 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:25.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:25.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:25.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:25.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:25.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:25.245 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:25.244+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:25.551 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:25.551+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:25.551 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:25.551+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:25.551 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:25.551+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:25.551 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:25.551+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:25.806 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f60 legacy=0x7f715c010780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:25.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.806+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:25.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:25.806+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:25.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:25.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:25.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:25.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:25.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:25.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:25.858 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:25.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:26.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:26.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:26.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:26.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:26.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:26.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:26.208 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:26.208+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:26.209 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:26.208+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:26.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:26.238+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:26.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:26.238+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:26.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:26.238+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:26.238 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:26.238+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:26.413 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:26.412+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5070 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:26.412+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c0137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:45:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:26.412+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c0137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:45:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:26.412+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 0x7f715c0137e0 :-1 s=OPENED pgs=3238 cs=1 l=1).handle_message read tag failed 2023-07-04T23:45:26.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:26.412+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 0x7f715c0137e0 :-1 s=OPENED pgs=3238 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:45:26.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:26.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:26.594 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:26.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:26.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:26.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:26.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:26.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:26.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:26.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:26.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:26.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:26.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:26.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:26.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:26.892+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:27.193 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:27.193+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:27.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:27.193+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:27.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:27.193+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:27.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:27.193+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:27.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:27.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:27.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:27.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:27.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:27.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:27.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:27.238+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:27.238 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:27.238+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013ef0 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f60 legacy=0x7f715c010780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00d390 legacy=0x7f715c0137e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:27.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0091b0 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0088a0 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:27.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:27.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:27.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.404+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:27.597 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:27.597+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:27.598 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:27.597+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:27.598 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:27.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:27.598 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:27.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:27.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:27.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:27.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:27.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:27.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:27.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:27.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:27.605+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:27.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:27.927+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:27.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:27.927+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:27.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:27.927+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:27.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:27.927+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:28.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:28.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.006+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:28.174 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:28.174+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:28.174 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:28.174+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:28.174 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:28.174+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:28.174 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:28.174+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:28.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:28.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:28.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:28.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:28.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:28.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:28.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:28.278+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:28.279 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:28.278+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:28.607 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:28.606+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:28.607 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:28.606+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:28.608 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:28.606+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:28.608 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:28.606+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:28.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:28.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:28.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:28.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:28.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:28.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:28.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:28.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:28.897+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:28.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:28.897+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:28.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:28.897+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:28.899 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:28.897+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:29.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:29.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:29.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:29.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:29.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:29.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:29.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:29.125+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:29.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:29.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:29.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:29.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:29.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:29.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:29.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:29.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:29.316 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:29.314+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:29.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:29.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:29.595 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:29.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:29.596 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:29.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:29.596 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:29.594+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:29.907 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:29.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:29.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:29.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:29.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:29.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:29.908 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:29.906+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:30.102 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:30.101+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:30.102 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:30.101+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:30.102 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:30.101+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:30.102 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:30.101+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:30.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:30.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:30.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:30.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:30.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:30.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:30.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:30.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:30.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:30.352+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:30.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:30.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:30.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0091b0 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0088a0 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0137b0 con 0x7f715c009f30 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:30.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:30.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.403+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:30.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:30.583+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:30.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:30.583+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:30.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:30.583+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:30.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:30.583+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:30.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:30.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:30.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:30.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:30.604+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:30.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:30.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:30.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:30.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:30.891 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:30.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:30.892 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:30.890+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:31.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:31.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:31.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:31.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:31.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:31.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:31.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:31.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:31.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.005+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:31.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:31.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:31.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:31.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:31.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:31.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:31.067 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:31.066+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:31.379 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:31.378+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:31.379 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:31.378+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:31.379 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:31.378+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:31.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:31.378+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:31.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:31.378+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:31.413 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:31.413+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5145 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:31.587 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:31.587+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:31.588 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:31.587+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:31.588 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:31.587+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:31.588 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:31.587+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:31.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:31.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:31.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:31.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:31.857 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:31.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:31.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:31.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:31.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:31.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:31.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:31.856+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:32.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:32.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:32.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:32.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:32.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:32.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:32.024 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:32.023+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:32.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:32.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:32.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:32.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:32.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:32.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:32.392 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:32.391+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:32.393 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:32.391+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:32.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:32.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:32.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:32.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:32.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:32.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:32.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:32.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:32.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:32.859+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:32.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:32.859+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:32.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:32.859+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:32.860 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:32.859+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:33.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:33.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:33.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:33.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:33.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:33.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:33.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:33.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:33.404 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c013270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c013270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 msgr2=0x7f715c008c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0088a0 0x7f715c008c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 msgr2=0x7f715c0095b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0091b0 0x7f715c0095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0144e0 con 0x7f715c013d90 2023-07-04T23:45:33.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c0139c0 2023-07-04T23:45:33.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.403+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c0139c0 legacy=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:33.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:33.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:33.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:33.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:33.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:33.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:33.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:33.434+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:33.435 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:33.434+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:33.555 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:33.554+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:33.555 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:33.554+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:33.555 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:33.554+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:33.555 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:33.554+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:33.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:33.604+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c0139c0 legacy=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:33.815 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:33.814+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:33.815 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:33.814+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:33.815 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:33.814+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:33.816 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:33.814+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:34.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:34.005+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c0139c0 legacy=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:34.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:34.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:34.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:34.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:34.047 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:34.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:34.048 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:34.046+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:34.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:34.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:34.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:34.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:34.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:34.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:34.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:34.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:34.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:34.437+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:34.573 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:34.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:34.573 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:34.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:34.574 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:34.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:34.574 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:34.573+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:34.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:34.806+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c0139c0 legacy=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:34.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:34.841+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:34.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:34.841+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:34.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:34.841+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:34.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:34.841+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:35.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:35.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:35.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:35.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:35.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:35.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:35.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:35.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:35.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:35.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:35.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:35.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:35.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:35.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:35.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:35.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:35.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:35.479+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:35.575 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:35.574+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:35.576 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:35.574+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:35.576 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:35.574+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:35.576 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:35.574+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:35.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:35.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:35.862 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:35.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:35.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:35.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:35.863 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:35.861+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:36.064 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:36.063+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:36.065 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:36.063+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:36.065 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:36.063+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:36.065 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:36.063+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c0139c0 legacy=0x7f715c007850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c013d90 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:36.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c007f60 con 0x7f715c0081a0 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0139c0 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c010ac0 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 msgr2=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.403+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.404+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.404+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:36.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.404+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:36.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.404+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:36.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.404+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:36.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.414+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 legacy=0x7f715c0103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-04T23:45:36.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.414+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 legacy=0x7f715c0103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:45:36.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.414+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 0x7f715c0103b0 :-1 s=OPENED pgs=3255 cs=1 l=1).handle_message read tag failed 2023-07-04T23:45:36.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.414+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 0x7f715c0103b0 :-1 s=OPENED pgs=3255 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:45:36.415 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:36.413+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5230 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:36.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:36.463+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:36.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:36.463+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:36.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:36.463+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:36.465 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:36.463+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:36.465 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:36.463+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:36.536 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:36.535+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:36.536 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:36.535+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:36.536 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:36.535+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:36.537 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:36.535+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:36.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.604+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 msgr2=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:36.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.604+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:36.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.605+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:36.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.605+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:36.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.605+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:36.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.605+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:36.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:36.605+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:36.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:36.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:36.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:36.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:36.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:36.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:36.859 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:36.858+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:37.005 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.005+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 msgr2=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:37.006 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.005+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:37.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.006+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:37.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.006+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:37.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.006+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:37.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.006+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:37.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.006+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:37.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:37.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:37.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:37.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:37.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:37.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:37.059 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:37.058+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:37.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:37.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:37.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:37.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:37.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:37.464 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:37.462+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:37.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:37.499+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:37.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:37.499+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:37.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:37.499+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:37.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:37.499+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:37.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.806+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 msgr2=0x7f715c007850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:37.807 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.806+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.807+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.807+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.807+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.807+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:37.807+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:37.889 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:37.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:37.890 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:37.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:37.890 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:37.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:37.890 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:37.889+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:38.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:38.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:38.017 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:38.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:38.018 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:38.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:38.018 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:38.016+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:38.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:38.443+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:38.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:38.443+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:38.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:38.443+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:38.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:38.443+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:38.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:38.443+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:38.535 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:38.534+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:38.535 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:38.534+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:38.536 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:38.534+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:38.536 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:38.534+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:38.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:38.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:38.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:38.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:38.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:38.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:38.936 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:38.934+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:39.032 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:39.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:39.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:39.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:39.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:39.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:39.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:39.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:39.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 msgr2=0x7f715c007850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:39.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010ac0 0x7f715c007850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:39.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 msgr2=0x7f715c008570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:39.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139c0 0x7f715c008570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:39.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 legacy=0x7f715c0103b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c009f30 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013270 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c008ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 msgr2=0x7f715c0091c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:39.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.404+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:39.461 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:39.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:39.461 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:39.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:39.461 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:39.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:39.460+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:39.462 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:39.460+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:39.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:39.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:39.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:39.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:39.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:39.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:39.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:39.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:39.605 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.605+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c008ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:39.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.605+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 msgr2=0x7f715c0091c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:39.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:39.605+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:39.935 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:39.935+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:39.936 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:39.935+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:39.936 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:39.935+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:39.936 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:39.935+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:40.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.006+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c008ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:40.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.006+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 msgr2=0x7f715c0091c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:40.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.006+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:40.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:40.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:40.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:40.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.025+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:40.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:40.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:40.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:40.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:40.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:40.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:40.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:40.437+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:40.438 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:40.437+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:40.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:40.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:40.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:40.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:40.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:40.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:40.599 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:40.598+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.807+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c008ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.807+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 msgr2=0x7f715c0091c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:40.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:40.807+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:40.975 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:40.975+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:40.976 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:40.975+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:40.976 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:40.975+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:40.976 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:40.975+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:40.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:40.985 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:40.986 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:40.986 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:40.985+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:41.367 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:45:41.370 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:45:41.405 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:41.404+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:41.405 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:41.404+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:41.405 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:41.404+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:41.405 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:41.404+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:41.406 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:41.404+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:41.415 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:41.414+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5308 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:41.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:41.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:41.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:41.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:41.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:41.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:41.589 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:41.588+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:41.960 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:41.959+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:41.960 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:41.959+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:41.960 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:41.959+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:41.961 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:41.959+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:42.018 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.019 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.019 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.019 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:42.405 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 msgr2=0x7f715c0091c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c013270 0x7f715c0091c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c008ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c009700 con 0x7f715c0081a0 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c009f30 2023-07-04T23:45:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c010ac0 2023-07-04T23:45:42.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:42.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:42.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.404+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:42.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:42.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:42.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:42.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:42.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:42.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:42.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:42.454+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:42.455 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:42.454+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:42.546 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:42.546+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:42.547 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:42.546+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:42.547 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:42.546+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:42.547 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:42.546+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.605+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.605+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:42.605+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:42.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.986+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.986+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.986+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:42.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:42.986+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:42.992 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:42.992+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:42.992 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:42.992+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:42.993 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:42.992+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:42.993 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:42.992+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.006+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.006+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.006+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:43.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:43.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:43.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:43.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:43.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:43.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:43.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:43.462+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:43.463 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:43.462+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:43.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:43.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:43.584 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:43.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:43.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:43.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:43.585 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:43.584+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.807+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.807+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:43.807+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:43.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:43.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:43.939 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:43.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:43.940 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:43.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:43.940 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:43.938+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:43.998 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:43.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:43.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:43.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:43.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:43.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:43.999 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:43.998+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:44.485 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:44.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:44.485 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:44.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:44.485 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:44.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:44.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:44.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:44.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:44.484+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:44.603 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:44.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:44.603 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:44.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:44.603 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:44.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:44.603 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:44.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:44.956 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:44.956+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:44.957 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:44.956+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:44.957 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:44.956+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:44.957 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:44.956+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:44.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:44.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:44.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:44.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:44.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:44.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:44.978 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:44.977+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:45.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:45.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c010ac0 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:45.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:45.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 legacy=0x7f715c0103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0139a0 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c013270 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:45.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:45.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:45.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:45.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.405+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:45.521 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:45.520+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:45.521 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:45.520+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:45.521 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:45.520+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:45.522 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:45.520+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:45.522 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:45.520+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:45.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:45.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:45.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:45.606+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:45.625 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:45.625+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:45.625 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:45.625+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:45.626 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:45.625+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:45.626 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:45.625+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:45.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:45.925+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:45.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:45.925+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:45.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:45.925+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:45.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:45.925+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:45.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:45.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:45.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:45.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:45.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:45.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:45.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:45.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:46.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.007+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:46.416 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:46.415+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5387 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:46.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:46.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:46.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:46.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:46.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:46.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:46.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:46.479+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:46.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:46.479+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:46.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:46.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:46.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:46.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:46.622 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:46.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:46.623 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:46.622+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:46.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:46.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:46.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:46.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:46.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:46.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:46.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:46.808+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:46.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:46.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:46.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:46.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:46.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:46.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:46.922 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:46.921+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:46.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:46.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:46.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:46.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:46.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:46.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:46.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:46.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:47.483 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:47.482+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:47.483 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:47.482+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:47.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:47.482+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:47.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:47.482+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:47.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:47.482+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:47.648 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:47.648+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:47.662 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:47.648+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:47.662 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:47.648+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:47.663 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:47.648+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:47.884 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:47.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:47.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:47.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:47.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:47.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:47.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:47.884+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:47.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:47.939+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:47.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:47.939+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:47.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:47.939+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:47.940 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:47.939+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:48.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:48.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0139a0 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c009950 con 0x7f715c0081a0 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c009f30 2023-07-04T23:45:48.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013270 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:48.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.405+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:48.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:48.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:48.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:48.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:48.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:48.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:48.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:48.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:48.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:48.480+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:48.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:48.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:48.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:48.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:48.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:48.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:48.604 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:48.602+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:48.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:48.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:48.606+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:48.893 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:48.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:48.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:48.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:48.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:48.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:48.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:48.893+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:48.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:48.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:48.933 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:48.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:48.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:48.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:48.934 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:48.933+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:49.007 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.007+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:49.457 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:49.457+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:49.457 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:49.457+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:49.458 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:49.457+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:49.458 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:49.457+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:49.458 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:49.457+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:49.562 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:49.562+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:49.563 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:49.562+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:49.563 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:49.562+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:49.563 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:49.562+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:49.809 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:49.808+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:49.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:49.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:49.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:49.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:49.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:49.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:49.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:49.878+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:49.900 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:49.899+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:49.900 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:49.899+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:49.900 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:49.899+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:49.900 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:49.899+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:50.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:50.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:50.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:50.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:50.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:50.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:50.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:50.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:50.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:50.447+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:50.522 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:50.522+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:50.522 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:50.522+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:50.523 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:50.522+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:50.523 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:50.522+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:50.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:50.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:50.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:50.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:50.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:50.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:50.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:50.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:50.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:50.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:50.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:50.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:50.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:50.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:50.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:50.930+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:51.406 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 msgr2=0x7f715c007830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c013270 0x7f715c007830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0081a0 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:51.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c010980 2023-07-04T23:45:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0105b0 2023-07-04T23:45:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:51.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.406+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:51.418 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:51.417+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5473 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:51.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:51.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:51.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:51.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:51.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:51.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:51.481 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:51.480+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:51.482 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:51.480+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:51.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:51.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:51.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:51.515 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:51.515+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.607+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.607+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.607+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:51.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:51.607+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:51.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:51.888+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:51.889 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:51.888+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:51.889 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:51.888+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:51.889 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:51.888+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:51.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:51.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:51.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:51.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:51.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:51.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:51.950 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:51.949+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:52.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.008+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:52.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.008+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:52.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.008+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:52.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.008+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:52.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:52.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:52.434 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:52.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:52.434 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:52.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:52.434 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:52.432+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:52.434 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:52.433+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:52.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:52.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:52.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:52.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:52.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:52.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:52.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:52.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:52.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.809+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:52.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.809+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:52.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.809+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:52.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:52.809+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:52.910 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:52.910+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:52.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:52.910+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:52.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:52.910+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:52.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:52.910+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:52.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:52.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:52.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:52.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:52.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:52.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:52.914 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:52.913+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:53.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:53.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:53.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:53.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:53.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:53.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:53.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:53.418+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:53.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:53.419+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:53.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:53.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:53.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:53.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:53.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:53.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:53.508 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:53.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:53.896 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:53.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:53.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:53.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:53.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:53.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:53.897 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:53.896+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:53.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:53.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:53.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:53.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:53.926 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:53.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:53.927 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:53.926+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:54.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0139a0 con 0x7f715c009f30 2023-07-04T23:45:54.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c010980 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0105b0 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:54.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:54.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.406+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:54.425 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:54.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:54.426 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:54.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:54.426 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:54.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:54.426 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:54.424+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:54.426 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:54.425+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:54.499 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:54.498+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:54.499 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:54.498+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:54.499 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:54.498+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:54.499 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:54.498+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:54.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:54.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:54.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:54.607+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:54.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:54.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:54.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:54.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:54.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:54.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:54.857 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:54.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:54.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:54.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:54.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:54.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:54.943 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:54.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:54.944 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:54.942+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:55.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:55.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:55.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:55.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:55.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:55.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:55.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:55.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.008+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:55.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:55.444+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:55.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:55.444+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:55.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:55.444+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:55.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:55.444+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:55.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:55.444+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:55.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:55.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:55.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:55.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:55.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:55.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:55.475 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:55.474+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:45:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:45:55.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:45:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:45:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:45:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:45:55.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:55.809+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:55.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:55.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:55.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:55.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:55.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:55.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:55.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:55.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:55.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:55.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:55.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:55.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:55.987 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:55.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:55.988 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:55.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:56.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:56.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:56.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:56.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:56.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:56.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:56.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:56.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:56.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:56.415+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:56.419 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:45:56.418+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5552 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:45:56.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:56.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:56.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:56.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:56.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:56.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:56.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:56.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:56.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:56.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:56.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:56.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:56.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:56.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:56.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:56.895+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:56.988 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:56.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:56.988 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:56.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:56.988 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:56.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:56.988 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:56.987+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:57.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0105b0 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 msgr2=0x7f715c00f8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c010980 0x7f715c00f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c014300 con 0x7f715c0091b0 2023-07-04T23:45:57.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013270 2023-07-04T23:45:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:57.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.406+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:45:57.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:57.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:57.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:57.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:57.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:57.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:57.448 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:57.447+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:57.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:57.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:57.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:57.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:57.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:57.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:57.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:57.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:57.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:57.449+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:57.607 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.607+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:57.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:57.607+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:45:57.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:57.894+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:57.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:57.894+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:57.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:57.894+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:57.895 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:57.894+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:57.999 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:57.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:57.999 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:57.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:57.999 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:57.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:57.999 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:57.999+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:58.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:58.008+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:58.008 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:58.008+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:45:58.415 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:58.415+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:58.415 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:58.415+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:58.415 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:58.415+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:58.415 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:58.415+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:58.454 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:58.453+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:58.454 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:58.453+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:58.454 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:58.453+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:58.454 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:58.453+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:58.454 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:58.453+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:58.809+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:45:58.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:45:58.809+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:45:58.876 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:58.876+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:58.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:58.876+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:58.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:58.876+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:58.877 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:58.876+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:59.041 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:59.041+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:59.041 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:59.041+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:59.042 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:59.041+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:45:59.042 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:45:59.041+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:45:59.422 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:59.423 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:45:59.423 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:59.423 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:45:59.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:59.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:59.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:45:59.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:45:59.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:45:59.422+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:45:59.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:59.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:59.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:59.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:45:59.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:59.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:45:59.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:45:59.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:00.074 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:00.073+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:00.074 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:00.073+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:00.074 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:00.073+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:00.075 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:00.073+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:00.393 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:00.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:00.393 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:00.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:00.393 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:00.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:00.393 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:00.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:00.407 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0091b0 legacy=0x7f715c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0139a0 con 0x7f715c009f30 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c007130 con 0x7f715c00fea0 2023-07-04T23:46:00.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013270 2023-07-04T23:46:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:00.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.407+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:00.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:00.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:00.439 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:00.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:00.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:00.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:00.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:00.439+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:00.440 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:00.439+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:00.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.608+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:00.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.608+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:00.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:00.608+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:00.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:00.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:00.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:00.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:00.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:00.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:00.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:00.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:01.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.009+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:01.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.009+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:01.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.009+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:01.072 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:01.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:01.072 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:01.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:01.072 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:01.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:01.072 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:01.072+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:01.378 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:01.378+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:01.379 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:01.378+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:01.379 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:01.378+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:01.379 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:01.378+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.419+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:46:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.419+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.419+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3298 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:01.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.419+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3298 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:01.421 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:01.419+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5628 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:01.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:01.467+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:01.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:01.467+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:01.468 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:01.467+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:01.469 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:01.467+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:01.469 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:01.467+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:01.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.810+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:01.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.810+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:01.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:01.810+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:01.825 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:01.825+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:01.825 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:01.825+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:01.825 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:01.825+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:01.826 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:01.825+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:02.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:02.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:02.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:02.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:02.108 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:02.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:02.109 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:02.108+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:02.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:02.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:02.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:02.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:02.408 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:02.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:02.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:02.408+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:02.512 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:02.511+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:02.512 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:02.511+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:02.512 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:02.511+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:02.512 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:02.511+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:02.512 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:02.511+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:02.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:02.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:02.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:02.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:02.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:02.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:02.862 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:02.861+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:03.157 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:03.157+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:03.158 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:03.157+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:03.158 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:03.157+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:03.158 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:03.157+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:03.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006730 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0109a0 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 msgr2=0x7f715c006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:03.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:03.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.407+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:03.438 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:03.437+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:03.438 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:03.437+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:03.438 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:03.437+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:03.439 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:03.437+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:03.528 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:03.528+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:03.528 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:03.528+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:03.528 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:03.528+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:03.528 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:03.528+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:03.529 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:03.528+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:03.608 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 msgr2=0x7f715c006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:03.609 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:03.608+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:03.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:03.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:03.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:03.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:03.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:03.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:03.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:03.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 msgr2=0x7f715c006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:04.009 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:04.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:04.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:04.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:04.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.009+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:04.174 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:04.174 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:04.174 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:04.175 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:04.174+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:04.402 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:04.402+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:04.402 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:04.402+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:04.402 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:04.402+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:04.403 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:04.402+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:04.523 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:04.522+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:04.523 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:04.522+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:04.523 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:04.522+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:04.523 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:04.522+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:04.524 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:04.522+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:04.810 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 msgr2=0x7f715c006120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:04.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:04.810+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:04.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:04.886+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:04.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:04.886+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:04.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:04.886+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:04.886 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:04.886+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:05.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:05.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:05.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:05.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:05.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:05.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:05.140 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:05.139+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:05.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:05.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:05.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:05.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:05.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:05.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:05.421 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:05.421+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:05.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:05.477+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:05.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:05.477+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:05.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:05.477+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:05.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:05.477+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:05.477 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:05.477+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:05.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:05.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:05.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:05.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:05.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:05.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:05.894 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:05.893+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:06.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:06.103+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:06.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:06.103+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:06.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:06.103+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:06.103 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:06.103+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:06.392 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:06.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:06.392 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:06.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:06.392 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:06.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:06.392 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:06.392+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 msgr2=0x7f715c006120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0109a0 0x7f715c006120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:06.408 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c013b70 con 0x7f715c009f30 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c007130 con 0x7f715c00fea0 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c013270 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:06.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:06.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.408+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:06.420 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.420+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:46:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.420+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.420+0000 7f717b7fe700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3307 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:06.421 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.420+0000 7f717b7fe700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3307 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:06.421 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:06.420+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5717 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:06.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:06.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:06.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:06.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:06.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:06.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:06.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:06.431+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:06.433 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:06.431+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:06.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.609+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:06.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.609+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:06.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:06.609+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:06.911 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:06.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:06.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:06.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:06.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:06.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:06.912 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:06.911+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:07.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.010+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:07.010 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.010+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:07.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.010+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:07.111 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:07.111+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:07.111 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:07.111+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:07.111 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:07.111+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:07.112 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:07.111+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:07.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:07.409+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:07.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:07.409+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:07.409 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:07.409+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:07.410 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:07.409+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:07.466 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:07.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:07.466 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:07.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:07.467 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:07.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:07.467 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:07.466+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:07.467 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:07.466+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.811+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.811+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:07.811+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:07.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:07.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:07.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:07.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:07.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:07.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:07.880 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:07.880+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:08.105 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:08.105+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:08.105 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:08.105+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:08.105 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:08.105+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:08.105 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:08.105+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:08.424 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:08.424+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:08.424 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:08.424+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:08.424 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:08.424+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:08.424 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:08.424+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:08.488 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:08.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:08.488 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:08.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:08.489 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:08.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:08.489 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:08.488+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:08.489 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:08.488+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:08.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:08.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:08.881 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:08.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:08.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:08.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:08.882 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:08.881+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:09.062 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:09.062+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:09.062 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:09.062+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:09.062 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:09.062+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:09.063 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:09.062+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:09.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:09.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c013270 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:09.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c00fea0 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006730 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c00e3a0 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 msgr2=0x7f715c00e770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:09.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:09.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:09.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:09.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:09.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:09.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:09.461 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:09.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:09.461 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:09.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:09.461 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:09.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:09.461 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:09.461+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:09.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:09.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:09.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:09.484+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:09.484+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 msgr2=0x7f715c00e770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:09.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:09.891 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:09.891+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:09.891 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:09.891+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:09.891 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:09.891+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:09.891 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:09.891+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 msgr2=0x7f715c00e770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:10.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:10.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:10.063 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:10.063+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:10.063 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:10.063+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:10.063 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:10.063+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:10.063 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:10.063+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:10.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:10.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:10.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:10.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:10.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:10.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:10.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:10.449+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:10.450 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:10.449+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:10.506 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:10.506+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:10.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:10.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:10.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:10.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:10.507 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:10.507+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 msgr2=0x7f715c00e770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:10.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:10.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:10.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:10.883 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:10.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:10.883 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:10.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:10.883 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:10.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:10.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:10.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:11.025 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:11.025+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:11.025 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:11.025+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:11.025 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:11.025+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:11.025 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:11.025+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:11.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:11.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:11.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:11.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:11.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:11.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:11.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:11.419+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:11.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:11.419+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:11.422 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:11.422+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5794 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:11.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:46:11.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:46:11.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:11.543+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:11.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:11.543+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:11.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:11.543+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:11.543 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:11.543+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:11.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:11.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:11.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:11.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:11.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:11.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:11.841 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:11.840+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:12.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.003+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.003+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.003+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.003+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:12.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 msgr2=0x7f715c00e770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:12.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00e3a0 0x7f715c00e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:12.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.407+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.407+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006730 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006120 con 0x7f715c009f30 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006730 2023-07-04T23:46:12.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c00fea0 2023-07-04T23:46:12.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:12.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:12.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:12.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.408+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:12.431 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:12.430+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:12.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:12.430+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:12.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:12.430+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:12.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:12.430+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:12.432 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:12.431+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:12.500 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:12.499+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:12.501 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:12.500+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:12.501 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:12.500+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:12.501 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:12.500+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:12.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.609+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:12.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.609+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:12.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.609+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:12.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:12.609+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:12.860 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:12.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:12.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:12.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:12.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:12.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:12.861 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:12.860+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:12.975 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.974+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.975 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.974+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.975 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.974+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:12.975 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:12.974+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.010+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.010+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.010+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.010+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:13.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:13.429+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:13.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:13.429+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:13.430 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:13.429+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:13.431 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:13.429+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:13.431 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:13.429+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:13.469 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:13.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:13.469 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:13.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:13.470 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:13.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:13.470 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:13.468+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.811+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.811+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.811+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:13.811+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:13.887 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:13.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:13.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:13.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:13.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:13.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:13.888 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:13.887+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:14.018 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.018 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.019 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.019 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.017+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:14.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:14.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:14.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:14.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:14.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:14.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:14.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:14.386+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:14.387 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:14.386+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:14.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:14.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:14.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:14.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:14.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:14.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:14.445 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:14.444+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:14.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:14.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:14.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:14.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:14.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:14.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:14.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:14.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:14.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.991 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:14.992 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:14.990+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:15.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:15.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:15.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:15.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:15.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:15.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:15.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:15.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:15.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:15.366+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:15.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:15.409 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c007130 con 0x7f715c013270 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006120 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:15.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:15.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:15.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:15.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:15.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.408+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:15.418 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:15.417+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:15.418 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:15.417+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:15.418 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:15.417+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:15.418 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:15.417+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:15.610 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:15.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:15.609+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:15.867 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:15.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:15.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:15.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:15.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:15.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:15.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:15.866+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:16.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:16.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:16.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:16.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:16.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:16.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.010+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:16.035 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:16.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:16.036 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:16.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:16.036 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:16.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:16.036 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:16.035+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:16.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:16.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:16.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:16.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:16.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:16.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:16.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:16.321+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:16.322 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:16.321+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:16.400 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:16.399+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:16.400 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:16.399+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:16.401 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:16.399+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:16.401 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:16.399+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:16.423 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:16.422+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5870 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:16.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:16.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:16.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:16.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:16.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:16.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:16.811+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:16.914 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:16.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:16.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:16.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:16.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:16.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:16.915 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:16.914+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:17.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:17.060+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:17.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:17.060+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:17.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:17.060+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:17.061 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:17.060+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:17.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:17.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:17.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:17.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:17.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:17.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:17.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:17.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:17.351 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:17.349+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:17.361 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:17.360+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:17.361 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:17.360+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:17.361 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:17.360+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:17.361 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:17.360+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:17.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:17.921+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:17.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:17.921+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:17.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:17.921+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:17.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:17.921+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:18.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:18.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:18.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:18.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:18.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:18.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:18.017 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:18.016+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:18.336 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:18.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:18.336 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:18.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:18.337 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:18.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:18.337 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:18.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:18.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:18.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:18.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:18.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:18.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:18.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:18.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:18.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:18.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:18.374+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:18.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:18.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:18.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:18.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:18.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00f8e0 con 0x7f715c013270 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006120 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.408+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c006730 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:18.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:18.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:18.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:18.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:18.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:18.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.409+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:18.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:18.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:18.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:18.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:18.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:18.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:18.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:18.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:18.610+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:18.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:18.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:18.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:18.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:18.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:18.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:18.904 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:18.903+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:19.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:19.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:19.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:19.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:19.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.011+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:19.024 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:19.023+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:19.024 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:19.023+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:19.024 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:19.023+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:19.024 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:19.023+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:19.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:19.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:19.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:19.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:19.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:19.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:19.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:19.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:19.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:19.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:19.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:19.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:19.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:19.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:19.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:19.413+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:19.414 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:19.413+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:19.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:19.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:19.812+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:19.942 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:19.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:19.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:19.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:19.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:19.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:19.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:19.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:20.041 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:20.040+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:20.041 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:20.040+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:20.042 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:20.040+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:20.042 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:20.040+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:20.283 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:20.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:20.283 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:20.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:20.283 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:20.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:20.283 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:20.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:20.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:20.448+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:20.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:20.448+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:20.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:20.448+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:20.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:20.448+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:20.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:20.448+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:20.905 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:20.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:20.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:20.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:20.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:20.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:20.906 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:20.904+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:21.059 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:21.059+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:21.060 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:21.059+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:21.060 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:21.059+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:21.060 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:21.059+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:21.306 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:21.305+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:21.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:21.305+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:21.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:21.305+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:21.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:21.305+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:21.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:21.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:21.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:21.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:21.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c010270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:21.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:21.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c007130 con 0x7f715c009f30 2023-07-04T23:46:21.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00fea0 2023-07-04T23:46:21.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:21.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.409+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:21.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:21.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:21.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:21.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:21.424 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:21.424 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:46:21.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:21.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3332 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:21.425 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.423+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3332 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:21.425 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:21.424+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 5955 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.610+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:21.610+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:21.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:21.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:21.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:21.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:21.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:21.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:21.875 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:21.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:22.011 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:22.011+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:22.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:22.011+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:22.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:22.097+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:22.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:22.097+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:22.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:22.097+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:22.098 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:22.097+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:22.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:22.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:22.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:22.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:22.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:22.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:22.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:22.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:22.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:22.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:22.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:22.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:22.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:22.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:22.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:22.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:22.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:22.374+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:22.812 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:22.812+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:22.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:22.812+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:22.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:22.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:22.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:22.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:22.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:22.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:22.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:22.845+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:23.078 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:23.077+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:23.078 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:23.077+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:23.078 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:23.077+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:23.079 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:23.077+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:23.285 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:23.284+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:23.285 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:23.284+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:23.285 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:23.284+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:23.285 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:23.284+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:23.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:23.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:23.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:23.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:23.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:23.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:23.396 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:23.395+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:23.397 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:23.395+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:23.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:23.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:23.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:23.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:23.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:24.078 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:24.078+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:24.079 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:24.078+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:24.079 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:24.078+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:24.079 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:24.078+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:24.307 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:24.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:24.308 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:24.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:24.308 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:24.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:24.308 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:24.306+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:24.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 msgr2=0x7f715c006b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:24.410 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fea0 0x7f715c006b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00f8e0 con 0x7f715c009f30 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006120 2023-07-04T23:46:24.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c006730 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:24.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.409+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:24.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:24.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:24.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:24.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:24.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:24.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:24.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:24.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:24.444 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:24.443+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:24.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:24.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:24.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:24.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:24.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:24.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:24.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:24.610+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:24.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:24.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:24.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:24.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:24.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:24.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:24.874 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:24.873+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:25.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:25.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:25.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:25.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:25.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:25.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:25.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.011+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:25.053 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:25.052+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:25.053 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:25.052+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:25.053 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:25.052+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:25.053 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:25.052+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:25.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:25.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:25.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:25.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:25.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:25.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:25.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:25.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:25.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:25.420+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:25.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:25.420+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:25.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:25.420+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:25.421 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:25.420+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:25.422 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:25.420+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:25.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:25.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:25.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:25.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:25.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:25.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:25.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:25.812+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:25.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:25.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:25.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:25.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:25.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:25.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:25.890 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:25.889+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:26.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:26.088+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:26.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:26.088+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:26.089 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:26.088+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:26.090 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:26.088+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:26.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:26.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:26.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:26.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:26.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:26.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:26.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:26.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:26.426 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:26.425+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6035 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:26.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:26.425+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-07-04T23:46:26.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:26.425+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:26.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:26.425+0000 7f7180899700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3341 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:26.425+0000 7f7180899700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 0x7f715c0077c0 :-1 s=OPENED pgs=3341 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:26.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:26.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:26.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:26.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:26.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:26.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:26.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:26.442+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:26.443 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:26.442+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:26.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:26.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:26.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:26.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:26.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:26.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:26.846 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:26.844+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:27.118 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:27.117+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:27.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:27.117+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:27.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:27.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:27.119 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:27.118+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:27.246 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:27.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:27.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:27.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:27.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:27.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:27.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:27.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:27.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:27.410 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:27.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:27.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:27.411 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:27.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 msgr2=0x7f715c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006730 0x7f715c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:27.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00fc10 con 0x7f715c013270 2023-07-04T23:46:27.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00fda0 2023-07-04T23:46:27.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:27.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 msgr2=0x7f715c010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:27.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.409+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:27.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.610+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 msgr2=0x7f715c010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:27.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.610+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:27.611 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:27.610+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:27.868 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:27.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:27.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:27.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:27.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:27.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:27.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:27.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:28.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.011+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:28.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.011+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 msgr2=0x7f715c010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:28.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.011+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:28.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:28.153+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:28.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:28.153+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:28.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:28.153+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:28.154 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:28.153+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:28.275 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:28.275+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:28.276 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:28.275+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:28.276 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:28.275+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:28.276 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:28.275+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:28.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:28.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:28.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:28.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:28.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:28.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:28.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:28.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:28.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:28.374+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:28.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.812+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 msgr2=0x7f715c010190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:28.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.812+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c006b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:28.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:28.812+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:28.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:28.819+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:28.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:28.819+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:28.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:28.819+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:28.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:28.819+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:29.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:29.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:29.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:29.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:29.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:29.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:29.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:29.163+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:29.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:29.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:29.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:29.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:29.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:29.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:29.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:29.268+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:29.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:29.342+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:29.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:29.342+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:29.342 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:29.342+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:29.343 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:29.342+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:29.343 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:29.342+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:29.800 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:29.800+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:29.801 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:29.800+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:29.801 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:29.800+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:29.801 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:29.800+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:30.125 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:30.125+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:30.126 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:30.125+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:30.126 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:30.125+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:30.126 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:30.125+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:30.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:30.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:30.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:30.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:30.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:30.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:30.242 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:30.241+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:30.323 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:30.322+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:30.324 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:30.322+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:30.324 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:30.322+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:30.324 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:30.322+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:30.324 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:30.322+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 msgr2=0x7f715c010190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c00fda0 0x7f715c010190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c013270 legacy=0x7f715c006b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0106d0 con 0x7f715c00fda0 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00fc10 con 0x7f715c009f30 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006120 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:30.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:30.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.410+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:30.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:30.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:30.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:30.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:30.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:30.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:30.611+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:30.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:30.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:30.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:30.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:30.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:30.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:30.845 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:30.843+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:31.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:31.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.012+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:31.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:31.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:31.134 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:31.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:31.135 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:31.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:31.135 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:31.134+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:31.253 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:31.253+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:31.254 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:31.253+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:31.254 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:31.253+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:31.254 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:31.253+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:31.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:31.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:31.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:31.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:31.371 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:31.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:31.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:31.370+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:31.372 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:31.370+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:31.426 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.425+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-07-04T23:46:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.425+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.425+0000 7f7180899700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 0x7f715c006730 :-1 s=OPENED pgs=3349 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.425+0000 7f7180899700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 0x7f715c006730 :-1 s=OPENED pgs=3349 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:31.427 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:31.426+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6110 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:31.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:31.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:31.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:31.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:31.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:31.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:31.813+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:31.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:31.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:31.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:31.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:31.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:31.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:31.885 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:31.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:32.163 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:32.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:32.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:32.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:32.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:32.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:32.164 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:32.162+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:32.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:32.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:32.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:32.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:32.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:32.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:32.293 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:32.292+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:32.418 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:32.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:32.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:32.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:32.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:32.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:32.419 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:32.417+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:32.420 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:32.417+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:32.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:32.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:32.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:32.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:32.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:32.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:32.884 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:32.883+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:33.177 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:33.177+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:33.178 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:33.177+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:33.178 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:33.177+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:33.178 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:33.177+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:33.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:33.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:33.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:33.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:33.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:33.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:33.296 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:33.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:33.377 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:33.376+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:33.377 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:33.376+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:33.377 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:33.376+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:33.378 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:33.376+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:33.378 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:33.376+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 msgr2=0x7f715c00f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c006120 0x7f715c00f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:33.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00e3a0 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c005bc0 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c00e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:33.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:33.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.410+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:33.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.611+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c00e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:33.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.611+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:33.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.611+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:33.612 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:33.611+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:33.892 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:33.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:33.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:33.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:33.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:33.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:33.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:33.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.012+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.012+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c00e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.012+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.012+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:34.149 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:34.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:34.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:34.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:34.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:34.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:34.150 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:34.149+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:34.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:34.256+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:34.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:34.256+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:34.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:34.256+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:34.257 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:34.256+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:34.412 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:34.412+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:34.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:34.412+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:34.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:34.412+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:34.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:34.412+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:34.413 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:34.412+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:34.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.813+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:34.813 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.813+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c00e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:34.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.813+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:34.814 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:34.813+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:34.922 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:34.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:34.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:34.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:34.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:34.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:34.923 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:34.922+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:35.171 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:35.170+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:35.171 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:35.170+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:35.172 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:35.170+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:35.172 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:35.170+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:35.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:35.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:35.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:35.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:35.223 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:35.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:35.224 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:35.223+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:35.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:35.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:35.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:35.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:35.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:35.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:35.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:35.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:35.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:35.366+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:35.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:35.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:35.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:35.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:35.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:35.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:35.879 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:35.878+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:36.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:36.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:36.219 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:36.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:36.220 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:36.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:36.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:36.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:36.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:36.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:36.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:36.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:36.270 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:36.269+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:36.357 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:36.357+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:36.357 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:36.357+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:36.357 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:36.357+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:36.358 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:36.357+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:36.358 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:36.357+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:36.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:36.411 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c00e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c00e3a0 0x7f715c00e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006e40 con 0x7f715c00fda0 2023-07-04T23:46:36.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c006120 2023-07-04T23:46:36.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c005bc0 2023-07-04T23:46:36.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:36.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:36.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:36.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.411+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:36.427 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.427+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:46:36.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.427+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:36.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.427+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 0x7f715c006730 :-1 s=OPENED pgs=3358 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:36.428 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.427+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 0x7f715c006730 :-1 s=OPENED pgs=3358 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:36.428 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:36.427+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6196 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.612+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.612+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.612+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:36.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:36.612+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:36.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:36.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:36.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:36.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:36.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:36.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:36.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:36.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:37.013 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.013+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:37.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.013+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:37.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.013+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:37.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.013+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:37.247 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:37.246+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:37.247 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:37.246+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:37.247 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:37.246+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:37.247 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:37.246+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:37.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:37.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:37.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:37.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:37.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:37.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:37.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:37.286+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:37.369 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:37.369+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:37.369 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:37.369+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:37.370 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:37.369+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:37.370 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:37.369+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:37.370 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:37.369+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:37.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.814+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:37.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.814+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:37.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.814+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:37.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:37.814+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:37.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:37.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:37.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:37.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:37.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:37.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:37.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:37.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:38.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:38.235+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:38.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:38.235+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:38.235 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:38.235+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:38.236 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:38.235+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:38.240 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:38.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:38.240 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:38.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:38.240 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:38.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:38.240 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:38.240+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:38.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:38.363+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:38.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:38.363+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:38.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:38.363+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:38.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:38.363+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:38.364 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:38.363+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:38.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:38.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:38.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:38.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:38.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:38.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:38.869 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:38.868+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:39.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:39.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:39.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:39.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:39.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:39.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:39.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:39.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:39.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:39.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:39.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:39.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:39.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:39.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:39.223 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:39.222+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:39.360 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:39.360+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:39.361 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:39.360+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:39.361 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:39.360+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:39.361 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:39.360+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:39.361 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:39.360+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:39.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 msgr2=0x7f715c010170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:39.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c005bc0 0x7f715c010170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:39.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 msgr2=0x7f715c0077c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:39.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c006120 0x7f715c0077c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:39.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00fda0 legacy=0x7f715c006730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00fc10 con 0x7f715c009f30 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.410+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00e3a0 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:39.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:39.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:39.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.411+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:39.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:39.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:39.612+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:39.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:39.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:39.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:39.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:39.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:39.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:39.853 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:39.852+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:40.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.013+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:40.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:40.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:40.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:40.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:40.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:40.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:40.212 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:40.211+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:40.243 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:40.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:40.244 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:40.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:40.244 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:40.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:40.244 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:40.243+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:40.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:40.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:40.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:40.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:40.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:40.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:40.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:40.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:40.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:40.374+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:40.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:40.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:40.814+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:40.827 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:40.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:40.827 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:40.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:40.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:40.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:40.828 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:40.827+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:41.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:41.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:41.198 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:41.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:41.199 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:41.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:41.199 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:41.197+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:41.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:41.204+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:41.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:41.204+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:41.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:41.204+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:41.204 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:41.204+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:41.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:41.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:41.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:41.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:41.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:41.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:41.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:41.352+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:41.353 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:41.352+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:41.429 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:41.428+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6273 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:41.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:46:41.673 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:46:41.823 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:41.823+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:41.824 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:41.823+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:41.824 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:41.823+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:41.824 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:41.823+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:42.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:42.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:42.185 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:42.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:42.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:42.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:42.186 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:42.185+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:42.219 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:42.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:42.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:42.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:42.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:42.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:42.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:42.219+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:42.347 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:42.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:42.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:42.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:42.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:42.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:42.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:42.347+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:42.348 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:42.347+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006e40 con 0x7f715c006120 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00fc10 con 0x7f715c009f30 2023-07-04T23:46:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00e3a0 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:42.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.411+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:42.612+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:42.797 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:42.797+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:42.798 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:42.797+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:42.798 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:42.797+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:42.798 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:42.797+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:43.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.013+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:43.220 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:43.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:43.221 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:43.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:43.221 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:43.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:43.221 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:43.219+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:43.231 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:43.231+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:43.232 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:43.231+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:43.232 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:43.231+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:43.232 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:43.231+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:43.349 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:43.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:43.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:43.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:43.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:43.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:43.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:43.349+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:43.350 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:43.349+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:43.755 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:43.755+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:43.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:43.755+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:43.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:43.755+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:43.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:43.755+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:43.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:43.814+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:44.231 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:44.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:44.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:44.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:44.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:44.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:44.232 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:44.231+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:44.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:44.273+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:44.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:44.273+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:44.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:44.273+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:44.273 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:44.273+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:44.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:44.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:44.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:44.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:44.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:44.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:44.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:44.365+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:44.366 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:44.365+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:44.758 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:44.757+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:44.758 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:44.757+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:44.758 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:44.757+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:44.758 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:44.757+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:45.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:45.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:45.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:45.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:45.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:45.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:45.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:45.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:45.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:45.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:45.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:45.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:45.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:45.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:45.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:45.294+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:45.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:45.382+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:45.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:45.382+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:45.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:45.382+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:45.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:45.382+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:45.382 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:45.382+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:45.412 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c006730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c006120 legacy=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0077c0 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0102e0 2023-07-04T23:46:45.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:45.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 msgr2=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:45.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:45.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.412+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.613+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.613+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 msgr2=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.613+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:45.613+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:45.783 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:45.782+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:45.783 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:45.782+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:45.783 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:45.782+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:45.784 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:45.782+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:46.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.014+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:46.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.014+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 msgr2=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:46.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:46.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.014+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:46.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:46.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:46.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:46.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:46.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:46.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:46.241 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:46.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:46.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:46.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:46.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:46.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:46.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:46.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:46.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:46.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:46.383 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:46.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:46.383 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:46.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:46.383 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:46.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:46.383 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:46.383+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:46.384 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:46.383+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:46.431 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:46.430+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6351 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:46.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:46.756+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:46.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:46.756+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:46.756 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:46.756+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:46.757 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:46.756+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:46.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.815+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:46.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.815+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 msgr2=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:46.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:46.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:46.815+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:47.275 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:47.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:47.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:47.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:47.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:47.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:47.276 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:47.275+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:47.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:47.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:47.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:47.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:47.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:47.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:47.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:47.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:47.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:47.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:47.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:47.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:47.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:47.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:47.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:47.416+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:47.417 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:47.416+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:47.774 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:47.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:47.774 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:47.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:47.774 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:47.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:47.774 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:47.774+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:48.290 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:48.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:48.290 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:48.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:48.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:48.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:48.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:48.289+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:48.325 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:48.325+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:48.325 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:48.325+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:48.325 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:48.325+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:48.325 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:48.325+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:48.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 msgr2=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:48.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0102e0 0x7f715c00d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:48.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:48.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0077c0 0x7f715c00fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:48.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00e3a0 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c009f30 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c0137b0 con 0x7f715c006120 2023-07-04T23:46:48.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:48.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:48.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.412+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:48.456 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:48.456+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:48.456 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:48.456+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:48.456 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:48.456+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:48.457 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:48.456+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:48.457 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:48.456+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:48.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:48.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:48.613+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:48.752 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:48.752+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:48.753 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:48.752+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:48.753 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:48.752+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:48.753 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:48.752+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:49.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:49.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:49.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:49.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:49.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:49.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:49.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:49.294 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:49.293+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:49.312 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:49.312+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:49.312 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:49.312+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:49.312 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:49.312+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:49.312 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:49.312+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:49.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:49.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:49.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:49.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:49.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:49.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:49.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:49.422+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:49.423 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:49.422+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:49.785 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:49.785+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:49.786 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:49.785+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:49.786 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:49.785+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:49.786 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:49.785+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:49.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f7180899700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:49.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:49.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:50.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:50.289+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:50.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:50.289+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:50.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:50.289+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:50.289 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:50.289+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:50.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:50.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:50.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:50.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:50.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:50.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:50.339 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:50.338+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:50.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:50.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:50.374 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:50.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:50.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:50.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:50.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:50.374+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:50.375 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:50.374+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:50.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:50.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:50.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:50.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:50.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:50.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:50.820 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:50.820+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:51.326 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:51.326+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:51.326 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:51.326+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:51.326 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:51.326+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:51.326 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:51.326+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:51.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:51.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:51.332 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:51.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:51.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:51.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:51.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:51.332+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:51.344 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:51.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:51.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:51.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:51.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:51.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:51.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:51.344+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:51.345 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:51.344+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c006120 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:51.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:51.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:51.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c00fc10 con 0x7f715c0077c0 2023-07-04T23:46:51.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c008630 con 0x7f715c005b50 2023-07-04T23:46:51.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.412+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c005b50 legacy=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:51.432 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:51.431+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6436 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:51.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.431+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0077c0 legacy=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-04T23:46:51.432 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.431+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0077c0 legacy=0x7f715c00fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:46:51.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.431+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0077c0 0x7f715c00fda0 :-1 s=OPENED pgs=3383 cs=1 l=1).handle_message read tag failed 2023-07-04T23:46:51.433 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.431+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0077c0 0x7f715c00fda0 :-1 s=OPENED pgs=3383 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:46:51.613 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:51.613+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c005b50 legacy=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:51.856 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:51.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:51.856 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:51.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:51.856 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:51.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:51.856 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:51.856+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:52.014 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:52.014+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c005b50 legacy=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:52.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:52.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:52.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:52.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:52.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:52.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:52.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:52.284+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:52.296 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:52.297+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:52.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:52.297+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:52.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:52.297+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:52.297 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:52.297+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:52.314 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:52.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:52.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:52.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:52.314+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:52.315 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:52.314+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:52.815 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:52.815+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c005b50 legacy=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:52.818 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:52.818+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:52.819 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:52.818+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:52.819 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:52.818+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:52.819 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:52.818+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:53.281 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:53.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:53.281 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:53.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:53.281 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:53.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:53.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:53.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:53.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:53.300+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:53.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:53.300+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:53.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:53.300+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:53.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:53.300+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:53.300 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:53.300+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:53.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:53.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:53.335 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:53.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:53.336 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:53.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:53.336 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:53.335+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:53.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:53.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:53.849 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:53.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:53.850 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:53.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:53.850 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:53.849+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:54.239 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:54.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:54.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:54.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:54.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:54.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:54.240 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:54.239+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:54.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:54.272+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:54.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:54.272+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:54.272 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:54.272+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:54.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:54.272+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:54.273 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:54.272+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:54.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:54.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:54.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:54.333 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:54.333+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c005b50 legacy=0x7f715c00d390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:54.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c0077c0 legacy=0x7f715c00fda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006c70 con 0x7f715c009f30 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c00e3a0 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0077c0 2023-07-04T23:46:54.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:54.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.413+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:54.614+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:54.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:54.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:54.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:54.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:54.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:54.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:54.893 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:54.892+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:55.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:55.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:55.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:55.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:55.015 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:55.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:55.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:55.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.014+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:55.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:55.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:55.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:55.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:55.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:55.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:55.240 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:55.239+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:55.241 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:55.239+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:55.282 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:55.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:55.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:55.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:55.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:55.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:55.283 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:55.281+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:55.327 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:55.326+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:55.327 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:55.326+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:55.327 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:55.326+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:55.327 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:55.326+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:46:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:46:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:55.816 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:46:55.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:46:55.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:46:55.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:46:55.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:55.815+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:55.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:55.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:55.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:55.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:55.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:55.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:55.943 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:55.942+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:56.283 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:56.282+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:56.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:56.282+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:56.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:56.282+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:56.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:56.282+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:56.284 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:56.283+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:56.311 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:56.310+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:56.311 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:56.310+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:56.311 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:56.310+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:56.312 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:56.310+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:56.317 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:56.316+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:56.317 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:56.316+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:56.317 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:56.316+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:56.317 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:56.316+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:56.433 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:46:56.432+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6514 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:46:56.949 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:56.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:56.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:56.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:56.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:56.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:56.950 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:56.949+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:57.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:57.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:57.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:57.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:57.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:57.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:57.268 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:57.267+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:57.316 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:57.315+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:57.316 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:57.315+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:57.316 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:57.315+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:57.317 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:57.315+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:57.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:57.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:57.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:57.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:57.331 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:57.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:57.332 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:57.330+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:57.332 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:57.330+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:57.413 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 msgr2=0x7f715c008800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c00e3a0 0x7f715c008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c009f30 legacy=0x7f715c00fae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c008630 con 0x7f715c007b90 2023-07-04T23:46:57.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0101f0 2023-07-04T23:46:57.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:57.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.412+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:46:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.614+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:57.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:46:57.990 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:57.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:57.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:57.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:57.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:57.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:57.991 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:57.989+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:58.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:58.015+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:58.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:58.015+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:46:58.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:58.285+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:58.286 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:58.285+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:58.287 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:58.285+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:58.287 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:58.285+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:58.297 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:58.296+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:58.297 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:58.296+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:58.298 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:58.296+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:58.298 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:58.296+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:58.379 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:58.379+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:58.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:58.379+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:58.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:58.379+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:58.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:58.379+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:58.380 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:58.379+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:58.816+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:46:58.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:46:58.816+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:46:58.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:58.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:58.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:58.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:58.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:58.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:58.977 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:58.976+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:59.251 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:59.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:59.251 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:59.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:46:59.251 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:59.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:59.251 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:46:59.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:46:59.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:59.261+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:59.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:59.261+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:59.262 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:59.261+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:46:59.263 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:46:59.261+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:46:59.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:59.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:59.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:59.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:59.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:59.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:46:59.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:59.366+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:46:59.367 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:46:59.366+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:46:59.930 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:59.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:59.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:59.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:46:59.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:59.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:46:59.931 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:46:59.929+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:00.283 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:00.284 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:00.284 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:00.284 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:00.284 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:00.285 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:00.285 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:00.285 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:00.282+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:00.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:00.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:00.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:00.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:00.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:00.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:00.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:00.406+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:00.407 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:00.406+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:00.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.412+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101f0 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.412+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c007b90 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006c70 con 0x7f715c008800 2023-07-04T23:47:00.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c009f30 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c0077c0 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:00.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:00.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.413+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:00.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:00.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:00.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:00.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:00.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:47:00.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:00.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:00.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:00.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:00.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:00.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:00.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:00.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:00.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:00.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:00.957 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:00.956+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:01.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:01.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:01.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:01.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:01.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:47:01.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:01.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:01.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.015+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:01.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:01.288+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:01.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:01.288+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:01.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:01.288+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:01.289 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:01.288+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:01.304 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:01.303+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:01.304 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:01.303+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:01.304 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:01.303+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:01.304 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:01.303+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:01.415 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:01.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:01.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:01.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:01.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:01.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:01.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:01.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:01.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:01.415+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:01.434 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:47:01.433+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6589 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:47:01.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:01.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:01.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:01.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:01.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-07-04T23:47:01.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:01.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:01.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:01.816+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:01.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:01.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:01.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:01.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:01.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:01.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:01.952 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:01.951+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:02.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:02.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:02.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:02.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:02.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:02.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:02.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:02.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:02.323 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:02.322+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:02.323 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:02.322+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:02.323 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:02.322+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:02.324 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:02.322+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:02.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:02.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:02.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:02.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:02.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:02.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:02.368 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:02.367+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:02.369 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:02.367+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:02.988 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:02.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:02.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:02.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:02.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:02.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:02.989 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:02.988+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:03.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:03.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:03.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:03.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:03.247 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:03.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:03.248 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:03.246+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:03.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:03.305+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:03.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:03.305+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:03.306 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:03.305+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:03.307 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:03.305+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:03.343 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:03.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:03.344 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:03.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:03.344 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:03.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:03.344 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:03.343+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:03.344 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:03.343+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 msgr2=0x7f715c0080f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c0077c0 0x7f715c0080f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:47:03.414 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 msgr2=0x7f715c006730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c009f30 0x7f715c006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c008630 con 0x7f715c00f360 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0101a0 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c018690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:03.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.413+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:03.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.614+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c018690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:03.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:03.614+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:04.015+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c018690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:04.016 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:04.015+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:04.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:04.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:04.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:04.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:04.033 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:04.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:04.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:04.032+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:04.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:04.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:04.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:04.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:04.213 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:04.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:04.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:04.212+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:04.337 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:04.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:04.337 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:04.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:04.338 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:04.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:04.338 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:04.336+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:04.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:04.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:04.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:04.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:04.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:04.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:04.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:04.388+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:04.389 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:04.388+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:04.816+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c018690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:04.816+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:05.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:05.048+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:05.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:05.048+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:05.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:05.048+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:05.049 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:05.048+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:05.191 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:05.190+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:05.191 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:05.190+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:05.191 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:05.190+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:05.191 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:05.190+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:05.365 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:05.364+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:05.365 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:05.364+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:05.365 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:05.364+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:05.365 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:05.364+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:05.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:05.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:05.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:05.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:05.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:05.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:05.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:05.415+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:05.416 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:05.415+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:06.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:06.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:06.093 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:06.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:06.094 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:06.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:06.094 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:06.092+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:06.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:06.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:06.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:06.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:06.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:06.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:06.190 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:06.189+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:06.398 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:06.397+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:06.399 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:06.397+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:06.399 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:06.397+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:06.399 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:06.397+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c018690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c018690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c00f360 legacy=0x7f715c00fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 reap_dead start 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:06.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006c70 con 0x7f715c008bd0 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c006fd0 con 0x7f715c008800 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.413+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0077c0 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:06.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:06.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.414+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:06.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.434+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c008bd0 legacy=0x7f715c01abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2023-07-04T23:47:06.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.434+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c008bd0 legacy=0x7f715c01abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-04T23:47:06.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.434+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c008bd0 0x7f715c01abe0 :-1 s=OPENED pgs=3411 cs=1 l=1).handle_message read tag failed 2023-07-04T23:47:06.435 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.434+0000 7f717bfff700 1 --1- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c008bd0 0x7f715c01abe0 :-1 s=OPENED pgs=3411 cs=1 l=1).fault on lossy channel, failing 2023-07-04T23:47:06.436 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:47:06.435+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6677 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:47:06.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:06.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:06.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:06.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:06.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:06.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:06.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:06.445+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:06.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:06.445+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:47:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:06.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:06.617 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:06.615+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.016+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:07.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:07.126+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:07.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:07.126+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:07.127 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:07.126+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:07.128 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:07.126+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:07.238 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:07.237+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:07.238 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:07.237+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:07.238 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:07.237+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:07.238 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:07.237+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:07.424 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:07.423+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:07.424 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:07.423+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:07.424 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:07.423+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:07.424 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:07.423+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:07.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:07.485+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:07.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:07.485+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:07.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:07.485+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:07.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:07.485+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:07.486 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:07.485+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f7180899700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.82:6789/0 2023-07-04T23:47:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-04T23:47:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-07-04T23:47:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-07-04T23:47:07.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-04T23:47:07.819 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:07.817+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:08.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:08.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:08.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:08.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:08.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:08.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:08.154 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:08.153+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:08.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:08.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:08.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:08.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:08.267 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:08.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:08.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:08.266+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:08.455 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:08.454+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:08.455 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:08.454+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:08.455 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:08.454+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:08.455 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:08.454+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:08.500 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:08.500+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:08.500 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:08.500+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:08.501 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:08.500+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:08.501 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:08.500+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:08.501 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:08.500+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:09.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:09.115+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:09.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:09.115+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:09.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:09.115+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:09.117 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:09.115+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:09.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:09.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:09.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:09.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:09.291 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:09.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:09.292 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:09.290+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:09.406 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:09.405+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:09.406 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:09.405+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:09.406 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:09.405+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:09.407 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:09.405+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 msgr2=0x7f715c0066d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.77:3300/0 conn(0x7f715c0077c0 0x7f715c0066d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-04T23:47:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.82:6789/0 conn(0x7f715c008800 legacy=0x7f715c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-04T23:47:09.415 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 >> v1:172.21.15.77:6789/0 conn(0x7f715c008bd0 legacy=0x7f715c01abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c00fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c00d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> v2:172.21.15.82:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717c0af2e0 con 0x7f715c0101a0 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f7179ffb700 1 -- 172.21.15.77:0/1941951922 --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c070d80 con 0x7f715c009f30 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c00fa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c00d290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:09.416 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c00d290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:09.417 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.414+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c00fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-04T23:47:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:09.483+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:09.483+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:09.483+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:09.484 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:09.483+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:09.485 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:09.483+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.615+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c00d290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.615+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c00fa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.615+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c00d290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:09.615+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c00fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-04T23:47:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.016+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c00fa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.016+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c00d290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.016+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c00fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:10.017 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.016+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c00d290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-04T23:47:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:10.121 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:10.120+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:10.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:10.267+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:10.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:10.267+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:10.268 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:10.267+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:10.269 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:10.267+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:10.438 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:10.437+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:10.438 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:10.437+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:10.438 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:10.437+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:10.439 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:10.437+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:10.494 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:10.493+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:10.494 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:10.493+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:10.494 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:10.493+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:10.494 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:10.493+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:10.494 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:10.493+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.817+0000 7f717bfff700 1 -- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 msgr2=0x7f715c00fa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3301/0 2023-07-04T23:47:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.817+0000 7f717b7fe700 1 -- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 msgr2=0x7f715c00d290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.82:3300/0 2023-07-04T23:47:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.817+0000 7f717b7fe700 1 --2- 172.21.15.77:0/1941951922 >> v2:172.21.15.82:3300/0 conn(0x7f715c0101a0 0x7f715c00d290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:10.818 INFO:teuthology.orchestra.run.smithi077.stderr:2023-07-04T23:47:10.817+0000 7f717bfff700 1 --2- 172.21.15.77:0/1941951922 >> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] conn(0x7f715c009f30 0x7f715c00fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-04T23:47:11.073 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:11.072+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:11.073 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:11.072+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:11.074 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:11.072+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:11.074 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:11.072+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:11.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:11.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:11.235 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:11.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:11.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:11.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:11.236 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:11.235+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:11.437 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:47:11.436+0000 7f6a7cb7c700 -1 mon.a@0(electing) e1 get_health_metrics reporting 6754 slow ops, oldest is mgrbeacon mgr.y(5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,4113, , 0) 2023-07-04T23:47:11.443 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:11.443+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:11.444 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:11.443+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:11.444 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:11.443+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:11.444 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:11.443+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:11.496 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:11.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:11.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:11.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:11.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:11.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:11.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:11.496+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:11.497 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:11.496+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:11.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:47:11.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:47:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-07-04T23:47:11.951 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-07-04T23:47:11.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T23:47:11.952 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-07-04T23:47:12.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T23:47:12.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T23:47:12.007 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-04T23:47:12.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:12.041+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:12.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:12.041+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:12.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:12.041+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:12.042 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:12.041+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:12.097 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-07-04T23:47:12.098 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T23:47:12.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20736 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 172.21.15.77:6789:/test /tmp/tmp.ujtNGq2JOmtest ceph rw,context=system_u:object_r:root_t:s0,relatime,name=admin,secret=,fsid=5e5bb10e-1ec2-4d2c-9b90-e60b54fdc4f5,acl 0 0 2023-07-04T23:47:12.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2023-07-04T23:47:12.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T23:47:12.117 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-07-04T23:47:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T23:47:12.175 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-07-04T23:47:12.175 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T23:47:12.175 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T23:47:12.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15938 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 2023-07-04T23:47:12.196 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T23:47:12.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T23:47:12.196 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-07-04T23:47:12.231 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:12.230+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:12.231 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:12.230+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:12.231 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:12.230+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:12.231 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:12.230+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:12.264 INFO:teuthology.orchestra.run.smithi082.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-07-04T23:47:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T23:47:12.265 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2023-07-04T23:47:12.266 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-04T23:47:12.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15938 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 2023-07-04T23:47:12.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi082.front.sepia.ceph.com 2023-07-04T23:47:12.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-04T23:47:12.325 DEBUG:teuthology.orchestra.run.smithi082:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-07-04T23:47:12.382 INFO:teuthology.orchestra.run.smithi082.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-07-04T23:47:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-04T23:47:12.435 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:12.434+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:12.435 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:12.434+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:12.435 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:12.434+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:12.435 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:12.434+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:12.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:12.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:12.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:12.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:12.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:12.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:12.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:12.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:12.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:12.535+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:13.044 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.043+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:13.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.043+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:13.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.043+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:13.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.043+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:13.188 INFO:tasks.ceph.osd.0:Sent signal 15 2023-07-04T23:47:13.189 INFO:tasks.ceph.osd.1:Sent signal 15 2023-07-04T23:47:13.189 INFO:tasks.ceph.osd.2:Sent signal 15 2023-07-04T23:47:13.190 INFO:tasks.ceph.osd.3:Sent signal 15 2023-07-04T23:47:13.190 INFO:tasks.ceph.mds.a:Sent signal 15 2023-07-04T23:47:13.190 INFO:tasks.ceph.mds.c:Sent signal 15 2023-07-04T23:47:13.191 INFO:tasks.ceph.mon.a:Sent signal 15 2023-07-04T23:47:13.191 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-07-04T23:47:13.191 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.188+0000 7f7dfbfa6700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 101150) UID: 0 2023-07-04T23:47:13.192 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.188+0000 7f7dfbfa6700 -1 osd.0 39 *** Got signal Terminated *** 2023-07-04T23:47:13.192 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.188+0000 7f7dfbfa6700 -1 osd.0 39 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-04T23:47:13.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f4f6a8d1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 101157) UID: 0 2023-07-04T23:47:13.192 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f4f6a8d1700 -1 osd.1 39 *** Got signal Terminated *** 2023-07-04T23:47:13.193 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f4f6a8d1700 -1 osd.1 39 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-04T23:47:13.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f9b4664e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 101161) UID: 0 2023-07-04T23:47:13.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f9b4664e700 -1 osd.2 39 *** Got signal Terminated *** 2023-07-04T23:47:13.193 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f9b4664e700 -1 osd.2 39 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-04T23:47:13.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f2bac08a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 101168) UID: 0 2023-07-04T23:47:13.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f2bac08a700 -1 osd.3 39 *** Got signal Terminated *** 2023-07-04T23:47:13.194 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:13.189+0000 7f2bac08a700 -1 osd.3 39 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-04T23:47:13.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f7bd7a51700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i a (PID: 114782) UID: 0 2023-07-04T23:47:13.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f7bd7a51700 -1 mds.a *** got signal Terminated *** 2023-07-04T23:47:13.195 INFO:tasks.ceph.mds.c.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f646347e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 114783) UID: 0 2023-07-04T23:47:13.195 INFO:tasks.ceph.mds.c.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f646347e700 -1 mds.c *** got signal Terminated *** 2023-07-04T23:47:13.195 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f6a7fb82700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 100836) UID: 0 2023-07-04T23:47:13.196 INFO:tasks.ceph.mon.a.smithi077.stderr:2023-07-04T23:47:13.190+0000 7f6a7fb82700 -1 mon.a@0(electing) e1 *** Got Signal Terminated *** 2023-07-04T23:47:13.210 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.209+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:13.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.209+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:13.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:13.211 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:13.210+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:13.392 INFO:tasks.ceph.mgr.y.smithi077.stderr:daemon-helper: command crashed with signal 15 2023-07-04T23:47:13.453 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:13.454 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:13.454 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:13.454 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:13.452+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:13.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:13.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:13.535 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:13.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.535+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:13.536 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:13.535+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:14.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:14.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:14.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:14.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:14.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:14.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:14.060 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:14.059+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:14.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:14.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:14.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:14.216 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:14.214+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:14.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:14.458+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:14.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:14.458+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:14.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:14.458+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:14.459 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:14.458+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:14.489+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:14.489+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:14.490 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:14.489+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:14.491 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:14.489+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:14.491 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:14.489+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:15.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:15.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:15.026 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:15.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:15.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:15.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:15.027 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:15.026+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:15.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:15.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:15.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:15.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:15.250 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:15.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:15.251 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:15.250+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:15.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:15.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:15.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:15.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:15.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:15.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:15.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:15.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:15.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:15.446+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:15.473 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:15.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:15.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:15.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:15.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:15.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:15.474 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:15.473+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:16.044 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:16.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:16.044 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:16.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:16.044 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:16.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:16.045 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:16.044+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:16.295 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:16.295+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:16.296 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:16.295+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:16.296 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:16.295+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:16.296 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:16.295+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:16.446 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:16.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:16.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:16.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:16.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:16.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:16.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:16.446+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:16.447 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:16.446+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:16.456 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:16.455+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:16.456 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:16.455+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:16.456 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:16.455+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:16.456 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:16.455+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:17.030 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:17.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:17.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:17.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:17.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:17.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:17.031 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:17.030+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:17.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:17.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:26.183327+0000 front 2023-07-04T23:36:12.081812+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:17.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:17.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.182447+0000 front 2023-07-04T23:36:16.182736+0000 (oldest deadline 2023-07-04T23:39:06.606516+0000) 2023-07-04T23:47:17.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:17.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.081929+0000 front 2023-07-04T23:36:12.084890+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:17.261 INFO:tasks.ceph.osd.0.smithi077.stderr:2023-07-04T23:47:17.260+0000 7f7df4b82700 -1 osd.0 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:12.081838+0000 front 2023-07-04T23:36:16.182517+0000 (oldest deadline 2023-07-04T23:39:14.108191+0000) 2023-07-04T23:47:17.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:17.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:12.126662+0000 front 2023-07-04T23:36:12.126849+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:17.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:17.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:12.126741+0000 front 2023-07-04T23:36:15.627071+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:17.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:17.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:12.126813+0000 front 2023-07-04T23:36:26.129106+0000 (oldest deadline 2023-07-04T23:39:13.752201+0000) 2023-07-04T23:47:17.448 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:17.447+0000 7f4f634ad700 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:26.129213+0000 front 2023-07-04T23:36:20.228381+0000 (oldest deadline 2023-07-04T23:39:27.253775+0000) 2023-07-04T23:47:17.449 INFO:tasks.ceph.osd.1.smithi077.stderr:2023-07-04T23:47:17.447+0000 7f4f634ad700 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.12:11869160 6.1 6:80b63aea:::100000234e0.00000582:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e39) 2023-07-04T23:47:17.451 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:17.450+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:13.049302+0000 front 2023-07-04T23:36:13.050463+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:17.452 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:17.450+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:17.650476+0000 front 2023-07-04T23:36:13.049314+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:17.452 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:17.450+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:17.650410+0000 front 2023-07-04T23:36:13.049406+0000 (oldest deadline 2023-07-04T23:39:17.575715+0000) 2023-07-04T23:47:17.452 INFO:tasks.ceph.osd.2.smithi077.stderr:2023-07-04T23:47:17.450+0000 7f9b3f22a700 -1 osd.2 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:13.050715+0000 front 2023-07-04T23:36:13.050635+0000 (oldest deadline 2023-07-04T23:40:12.284900+0000) 2023-07-04T23:47:18.034 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:18.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6804 osd.4 since back 2023-07-04T23:36:11.534440+0000 front 2023-07-04T23:36:26.133757+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:18.035 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:18.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6820 osd.5 since back 2023-07-04T23:36:16.232469+0000 front 2023-07-04T23:36:11.531944+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:18.035 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:18.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6812 osd.6 since back 2023-07-04T23:36:16.232411+0000 front 2023-07-04T23:36:26.133770+0000 (oldest deadline 2023-07-04T23:39:05.053718+0000) 2023-07-04T23:47:18.035 INFO:tasks.ceph.osd.3.smithi077.stderr:2023-07-04T23:47:18.034+0000 7f2ba4c66700 -1 osd.3 39 heartbeat_check: no reply from 172.21.15.82:6828 osd.7 since back 2023-07-04T23:36:11.534854+0000 front 2023-07-04T23:36:11.532042+0000 (oldest deadline 2023-07-04T23:39:15.054747+0000) 2023-07-04T23:47:41.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:47:41.978 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:48:12.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:48:12.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:48:42.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:48:42.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:49:12.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:49:12.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:49:42.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:49:42.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:50:12.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:50:12.725 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:50:42.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:50:42.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:51:13.083 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:51:13.086 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:51:43.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:51:43.209 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:52:13.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:52:13.340 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:52:43.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:52:43.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:53:13.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:53:13.519 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:53:43.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:53:43.674 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:54:13.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:54:13.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:54:43.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:54:43.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:55:14.072 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:55:14.074 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:55:44.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:55:44.213 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:56:14.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:56:14.353 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:56:44.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:56:44.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:57:14.570 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:57:14.573 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:57:44.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:57:44.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:58:14.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:58:14.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:58:44.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:58:44.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:59:15.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:59:15.071 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:59:45.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-04T23:59:45.192 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:00:15.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:00:15.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:00:45.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:00:45.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:01:15.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:01:15.471 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:01:45.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:01:45.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:02:15.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:02:15.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:02:45.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:02:45.854 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:03:15.993 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:03:15.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:03:46.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:03:46.141 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:04:16.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:04:16.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:04:46.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:04:46.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:05:16.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:05:16.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:05:46.704 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:05:46.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:06:16.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:06:16.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:06:47.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:06:47.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:07:17.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:07:17.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:07:47.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:07:47.308 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:08:17.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:08:17.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:08:47.575 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:08:47.578 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:09:17.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:09:17.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:09:47.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:09:47.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:10:18.049 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:10:18.052 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:10:48.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:10:48.179 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:11:18.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:11:18.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:11:48.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:11:48.457 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:12:18.590 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:12:18.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:12:48.709 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:12:48.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:13:18.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:13:18.839 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:13:48.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:13:48.983 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:14:19.115 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:14:19.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:14:49.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:14:49.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:15:19.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:15:19.397 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:15:49.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:15:49.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:16:19.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:16:19.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:16:49.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:16:49.820 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:17:19.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:17:19.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:17:50.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:17:50.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:18:20.231 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:18:20.234 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:18:50.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:18:50.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:19:20.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:19:20.514 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:19:50.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:19:50.650 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:20:20.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:20:20.796 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:20:50.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:20:50.929 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:21:21.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:21:21.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:21:51.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:21:51.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:22:21.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:22:21.360 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:22:51.501 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:22:51.504 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:23:21.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:23:21.655 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:23:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:23:51.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:24:21.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:24:21.966 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:24:52.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:24:52.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:25:22.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:25:22.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:25:52.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:25:52.400 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:26:22.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:26:22.544 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:26:52.693 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:26:52.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:27:22.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:27:22.843 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:27:52.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:27:52.991 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:28:23.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:28:23.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:28:53.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:28:53.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:29:23.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:29:23.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:29:53.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:29:53.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:30:23.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:30:23.727 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:30:53.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:30:53.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:31:24.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:31:24.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:31:54.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:31:54.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:32:24.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:32:24.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:32:54.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:32:54.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:33:24.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:33:24.581 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:33:54.731 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:33:54.734 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:34:24.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:34:24.877 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:34:55.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:34:55.003 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:35:25.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:35:25.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:35:55.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:35:55.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:36:25.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:36:25.393 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:36:55.499 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:36:55.502 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:37:25.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:37:25.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:37:55.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:37:55.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:38:25.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:38:25.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:38:55.946 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:38:55.949 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:39:26.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:39:26.064 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:39:56.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:39:56.129 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:40:26.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:40:26.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:40:56.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:40:56.449 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:41:26.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:41:26.730 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:41:56.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:41:56.993 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:42:27.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:42:27.182 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:42:57.322 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:42:57.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:43:27.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:43:27.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:43:57.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:43:57.625 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:44:27.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:44:27.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:44:57.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:44:57.977 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:45:28.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:45:28.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:45:58.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:45:58.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:46:28.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:46:28.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:46:58.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:46:58.579 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:47:28.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:47:28.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:47:58.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:47:58.841 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:48:28.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:48:28.993 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:48:59.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:48:59.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:49:29.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:49:29.270 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:49:59.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:49:59.408 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:50:29.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:50:29.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:50:59.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:50:59.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:51:29.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:51:29.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:51:59.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:51:59.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:52:29.885 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:52:29.887 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:53:00.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:53:00.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:53:30.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:53:30.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:54:00.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:54:00.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:54:30.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:54:30.410 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:55:00.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:55:00.541 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:55:30.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:55:30.698 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:56:00.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:56:00.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:56:30.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:56:30.967 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:57:01.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:57:01.082 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:57:31.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:57:31.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:58:01.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:58:01.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:58:31.462 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:58:31.465 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:59:01.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:59:01.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:59:31.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T00:59:31.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:00:01.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:00:01.750 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:00:31.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:00:31.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:01:02.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:01:02.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:01:32.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:01:32.092 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:02:02.245 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:02:02.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:02:32.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:02:32.412 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:03:02.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:03:02.560 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:03:32.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:03:32.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:04:02.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:04:02.912 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:04:33.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:04:33.017 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:05:03.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:05:03.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:05:33.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:05:33.222 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:06:03.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:06:03.361 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:06:33.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:06:33.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:07:03.794 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:07:03.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:07:33.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:07:33.944 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:08:04.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:08:04.120 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:08:34.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:08:34.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:09:04.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:09:04.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:09:34.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:09:34.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:10:04.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:10:04.606 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:10:34.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:10:34.736 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:11:04.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:11:04.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:11:35.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:11:35.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:12:05.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:12:05.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:12:35.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:12:35.284 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:13:05.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:13:05.348 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:13:35.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:13:35.490 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:14:05.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:14:05.558 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:14:35.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:14:35.710 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:15:05.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:15:05.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:15:36.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:15:36.019 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:16:06.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:16:06.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:16:36.304 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:16:36.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:17:06.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:17:06.445 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:17:36.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:17:36.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:18:06.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:18:06.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:18:36.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:18:36.852 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:19:06.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:19:06.978 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:19:37.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:19:37.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:20:07.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:20:07.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:20:37.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:20:37.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:21:07.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:21:07.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:21:37.780 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:21:37.783 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:22:07.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:22:07.925 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:22:38.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:22:38.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:23:08.176 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:23:08.179 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:23:38.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:23:38.356 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:24:08.506 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:24:08.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:24:38.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:24:38.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:25:08.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:25:08.779 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:25:38.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:25:38.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:26:09.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:26:09.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:26:39.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:26:39.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:27:09.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:27:09.438 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:27:39.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:27:39.598 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:28:09.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:28:09.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:28:39.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:28:39.902 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:29:10.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:29:10.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:29:40.184 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:29:40.187 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:30:10.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:30:10.333 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:30:40.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:30:40.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:31:10.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:31:10.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:31:40.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:31:40.756 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:32:10.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:32:10.883 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:32:41.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:32:41.027 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:33:11.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:33:11.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:33:41.329 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:33:41.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:34:11.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:34:11.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:34:41.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:34:41.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:35:11.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:35:11.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:35:41.845 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:35:41.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:36:11.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:36:11.992 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:36:42.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:36:42.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:37:12.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:37:12.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:37:42.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:37:42.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:38:12.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:38:12.558 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:38:42.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:38:42.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:39:12.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:39:12.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:39:43.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:39:43.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:40:13.156 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:40:13.159 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:40:43.290 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:40:43.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:41:13.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:41:13.423 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:41:43.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:41:43.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:42:13.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:42:13.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:42:43.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:42:43.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:43:13.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:43:13.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:43:44.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:43:44.109 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:44:14.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:44:14.257 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:44:44.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:44:44.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:45:14.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:45:14.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:45:44.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:45:44.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:46:14.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:46:14.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:46:44.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:46:44.961 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:47:15.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:47:15.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:47:45.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:47:45.247 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:48:15.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:48:15.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:48:45.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:48:45.519 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:49:15.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:49:15.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:49:45.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:49:45.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:50:16.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:50:16.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:50:46.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:50:46.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:51:16.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:51:16.304 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:51:46.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:51:46.473 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:52:16.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:52:16.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:52:46.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:52:46.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:53:16.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:53:16.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:53:47.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:53:47.083 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:54:17.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:54:17.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:54:47.367 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:54:47.369 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:55:17.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:55:17.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:55:47.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:55:47.698 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:56:17.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:56:17.842 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:56:47.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:56:48.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:57:18.151 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:57:18.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:57:48.287 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:57:48.290 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:58:18.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:58:18.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:58:48.579 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:58:48.582 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:59:18.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:59:18.738 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:59:48.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T01:59:48.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:00:19.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:00:19.051 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:00:49.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:00:49.203 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:01:19.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:01:19.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:01:49.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:01:49.491 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:02:19.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:02:19.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:02:49.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:02:49.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:03:19.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:03:19.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:03:50.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:03:50.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:04:20.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:04:20.144 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:04:50.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:04:50.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:05:20.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:05:20.457 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:05:50.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:05:50.599 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:06:20.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:06:20.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:06:50.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:06:50.924 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:07:21.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:07:21.079 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:07:51.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:07:51.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:08:21.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:08:21.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:08:51.693 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:08:51.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:09:21.844 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:09:21.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:09:51.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:09:51.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:10:22.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:10:22.113 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:10:52.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:10:52.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:11:22.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:11:22.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:11:52.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:11:52.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:12:22.882 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:12:22.885 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:12:53.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:12:53.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:13:23.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:13:23.180 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:13:53.317 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:13:53.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:14:23.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:14:23.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:14:53.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:14:53.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:15:23.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:15:23.738 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:15:53.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:15:53.891 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:16:24.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:16:24.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:16:54.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:16:54.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:17:24.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:17:24.305 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:17:54.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:17:54.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:18:24.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:18:24.676 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:18:54.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:18:54.828 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:19:24.969 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:19:24.972 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:19:55.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:19:55.107 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:20:25.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:20:25.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:20:55.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:20:55.384 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:21:25.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:21:25.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:21:55.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:21:55.678 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:22:25.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:22:25.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:22:55.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:22:55.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:23:26.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:23:26.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:23:56.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:23:56.268 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:24:26.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:24:26.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:24:56.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:24:56.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:25:26.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:25:26.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:25:56.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:25:56.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:26:26.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:26:27.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:26:57.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:26:57.153 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:27:27.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:27:27.307 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:27:57.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:27:57.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:28:27.590 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:28:27.593 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:28:57.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:28:57.730 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:29:27.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:29:27.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:29:57.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:29:58.001 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:30:28.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:30:28.137 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:30:58.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:30:58.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:31:28.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:31:28.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:31:58.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:31:58.583 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:32:28.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:32:28.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:32:58.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:32:58.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:33:29.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:33:29.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:33:59.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:33:59.157 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:34:29.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:34:29.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:34:59.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:34:59.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:35:29.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:35:29.574 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:35:59.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:35:59.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:36:30.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:36:30.027 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:37:00.177 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:37:00.181 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:37:30.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:37:30.299 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:38:00.431 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:38:00.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:38:30.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:38:30.570 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:39:00.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:39:00.714 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:39:30.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:39:30.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:40:00.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:40:00.975 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:40:31.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:40:31.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:41:01.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:41:01.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:41:31.399 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:41:31.402 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:42:01.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:42:01.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:42:31.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:42:31.670 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:43:01.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:43:01.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:43:31.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:43:31.979 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:44:02.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:44:02.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:44:32.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:44:32.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:45:02.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:45:02.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:45:32.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:45:32.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:46:02.725 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:46:02.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:46:32.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:46:32.858 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:47:03.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:47:03.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:47:33.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:47:33.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:48:03.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:48:03.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:48:33.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:48:33.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:49:03.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:49:03.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:49:33.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:49:33.733 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:50:03.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:50:03.869 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:50:34.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:50:34.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:51:04.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:51:34.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:51:34.304 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:52:04.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:52:04.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:52:34.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:52:34.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:53:04.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:53:04.747 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:53:34.897 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:53:34.900 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:54:04.958 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:54:04.960 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:54:35.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:54:35.112 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:55:05.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:55:05.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:55:35.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:55:35.391 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:56:05.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:56:05.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:56:35.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:56:35.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:57:05.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:57:05.832 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:57:35.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:57:35.974 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:58:06.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:58:06.131 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:58:36.251 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:58:36.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:59:06.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:59:06.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:59:36.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T02:59:36.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:00:06.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:00:06.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:00:36.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:00:36.985 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:01:07.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:01:07.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:01:37.248 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:01:37.251 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:02:07.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:02:07.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:02:37.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:02:37.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:03:07.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:03:07.699 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:03:37.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:03:37.851 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:04:08.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:04:08.091 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:04:38.245 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:04:38.248 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:05:08.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:05:08.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:05:38.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:05:38.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:06:08.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:06:08.698 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:06:38.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:06:38.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:07:08.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:07:08.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:07:39.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:07:39.115 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:08:09.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:08:09.278 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:08:39.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:08:39.404 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:09:09.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:09:09.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:09:39.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:09:39.746 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:10:09.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:10:09.886 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:10:40.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:10:40.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:11:10.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:11:10.163 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:11:40.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:11:40.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:12:10.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:12:10.486 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:12:40.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:12:40.622 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:13:10.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:13:10.758 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:13:40.881 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:13:40.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:14:11.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:14:11.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:14:41.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:14:41.172 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:15:11.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:15:11.315 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:15:41.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:15:41.484 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:16:11.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:16:11.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:16:41.773 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:16:41.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:17:11.938 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:17:11.941 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:17:42.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:17:42.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:18:12.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:18:12.304 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:18:42.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:18:42.438 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:19:12.563 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:19:12.566 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:19:42.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:19:42.718 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:20:12.872 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:20:12.876 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:20:43.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:20:43.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:21:13.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:21:13.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:21:43.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:21:43.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:22:13.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:22:13.498 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:22:43.634 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:22:43.637 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:23:13.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:23:13.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:23:43.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:23:43.921 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:24:14.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:24:14.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:24:44.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:24:44.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:25:14.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:25:14.327 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:25:44.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:25:44.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:26:14.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:26:14.644 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:26:44.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:26:44.787 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:27:14.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:27:14.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:27:45.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:27:45.100 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:28:15.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:28:15.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:28:45.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:28:45.362 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:29:15.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:29:15.506 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:29:45.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:29:45.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:30:15.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:30:15.766 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:30:45.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:30:45.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:31:16.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:31:16.055 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:31:46.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:31:46.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:32:16.303 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:32:16.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:32:46.375 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:32:46.378 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:33:16.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:33:16.441 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:33:46.506 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:33:46.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:34:16.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:34:16.569 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:34:46.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:34:46.655 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:35:16.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:35:16.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:35:46.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:35:46.812 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:36:16.942 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:36:16.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:36:47.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:36:47.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:37:17.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:37:17.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:37:47.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:37:47.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:38:17.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:38:17.472 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:38:47.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:38:47.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:39:17.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:39:17.751 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:39:48.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:39:48.114 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:40:18.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:40:18.267 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:40:48.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:40:48.417 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:41:18.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:41:18.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:41:48.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:41:48.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:42:18.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:42:18.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:42:48.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:42:48.781 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:43:18.853 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:43:18.856 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:43:48.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:43:48.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:44:19.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:44:19.049 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:44:49.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:44:49.139 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:45:19.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:45:19.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:45:49.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:45:49.416 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:46:19.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:46:19.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:46:49.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:46:49.672 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:47:19.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:47:19.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:47:49.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:47:49.969 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:48:20.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:48:20.113 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:48:50.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:48:50.239 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:49:20.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:49:20.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:49:50.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:49:50.535 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:50:20.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:50:20.674 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:50:50.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:50:50.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:51:20.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:51:20.994 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:51:51.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:51:51.156 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:52:21.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:52:21.321 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:52:51.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:52:51.464 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:53:21.605 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:53:21.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:53:51.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:53:51.744 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:54:21.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:54:21.871 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:54:51.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:54:51.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:55:22.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:55:22.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:55:52.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:55:52.252 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:56:22.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:56:22.387 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:56:52.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:56:52.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:57:22.630 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:57:22.634 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:57:52.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:57:52.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:58:22.942 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:58:22.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:58:53.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:58:53.010 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:59:23.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:59:23.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:59:53.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T03:59:53.272 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:00:23.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:00:23.388 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:00:53.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:00:53.507 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:01:23.623 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:01:23.626 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:01:53.767 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:01:53.770 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:02:23.910 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:02:23.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:02:54.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:02:54.065 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:03:24.191 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:03:24.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:03:54.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:03:54.345 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:04:24.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:04:24.481 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:04:54.612 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:04:54.615 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:05:24.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:05:24.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:05:54.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:05:54.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:06:25.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:06:25.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:06:55.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:06:55.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:07:25.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:07:25.301 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:07:55.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:07:55.436 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:08:25.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:08:25.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:08:55.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:08:55.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:09:25.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:09:25.835 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:09:55.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:09:55.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:10:26.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:10:26.124 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:10:56.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:10:56.312 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:11:26.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:11:26.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:11:56.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:11:56.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:12:26.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:12:26.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:12:56.929 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:12:56.932 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:13:27.070 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:13:27.073 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:13:57.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:13:57.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:14:27.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:14:27.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:14:57.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:14:57.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:15:27.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:15:27.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:15:57.775 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:15:57.778 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:16:27.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:16:27.905 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:16:58.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:16:58.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:17:28.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:17:28.276 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:17:58.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:17:58.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:18:28.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:18:28.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:18:58.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:18:58.717 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:19:28.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:19:28.864 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:19:59.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:19:59.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:20:29.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:20:29.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:20:59.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:20:59.327 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:21:29.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:21:29.459 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:21:59.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:21:59.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:22:29.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:22:29.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:22:59.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:22:59.865 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:23:30.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:23:30.008 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:24:00.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:24:00.162 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:24:30.328 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:24:30.330 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:25:00.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:25:00.468 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:25:30.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:25:30.618 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:26:00.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:26:00.753 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:26:30.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:26:30.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:27:01.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:27:01.066 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:27:31.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:27:31.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:28:01.422 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:28:01.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:28:31.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:28:31.556 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:29:01.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:29:01.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:29:31.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:29:31.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:30:01.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:30:01.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:30:32.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:30:32.090 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:31:02.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:31:02.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:31:32.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:31:32.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:32:02.484 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:32:02.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:32:32.789 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:32:32.792 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:33:02.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:33:02.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:33:33.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:33:33.027 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:34:03.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:34:03.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:34:33.303 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:34:33.306 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:35:03.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:35:03.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:35:33.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:35:33.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:36:03.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:36:03.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:36:33.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:36:33.872 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:37:04.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:37:04.016 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:37:34.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:37:34.152 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:38:04.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:38:04.304 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:38:34.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:38:34.455 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:39:04.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:39:04.600 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:39:34.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:39:34.736 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:40:04.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:40:04.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:40:35.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:40:35.107 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:41:05.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:41:05.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:41:35.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:41:35.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:42:05.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:42:05.438 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:42:35.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:42:35.512 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:43:05.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:43:05.576 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:43:35.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:43:35.655 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:44:05.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:44:05.829 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:44:35.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:44:35.973 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:45:06.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:45:06.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:45:36.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:45:36.270 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:46:06.425 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:46:06.428 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:46:36.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:46:36.597 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:47:06.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:47:06.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:47:36.864 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:47:36.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:48:06.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:48:06.988 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:48:37.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:48:37.119 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:49:07.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:49:07.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:49:37.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:49:37.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:50:07.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:50:07.536 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:50:37.664 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:50:37.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:51:07.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:51:07.789 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:51:37.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:51:37.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:52:08.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:52:08.067 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:52:38.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:52:38.238 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:53:08.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:53:08.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:53:38.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:53:38.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:54:08.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:54:08.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:54:38.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:54:38.503 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:55:08.564 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:55:08.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:55:38.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:55:38.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:56:08.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:56:08.830 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:56:38.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:56:38.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:57:09.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:57:09.067 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:57:39.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:57:39.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:58:09.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:58:09.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:58:39.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:58:39.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:59:09.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:59:09.609 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:59:39.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T04:59:39.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:00:09.893 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:00:09.896 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:00:40.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:00:40.024 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:01:10.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:01:10.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:01:40.299 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:01:40.302 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:02:10.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:02:10.379 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:02:40.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:02:40.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:03:10.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:03:10.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:03:40.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:03:40.835 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:04:10.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:04:10.987 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:04:41.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:04:41.123 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:05:11.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:05:11.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:05:41.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:05:41.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:06:11.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:06:11.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:06:41.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:06:41.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:07:11.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:07:11.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:07:41.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:07:41.986 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:08:12.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:08:12.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:08:42.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:08:42.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:09:12.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:09:12.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:09:42.499 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:09:42.503 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:10:12.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:10:12.630 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:10:42.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:10:42.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:11:12.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:11:12.895 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:11:43.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:11:43.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:12:13.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:12:13.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:12:43.368 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:12:43.371 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:13:13.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:13:13.500 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:13:43.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:13:43.632 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:14:13.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:14:13.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:14:43.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:14:43.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:15:14.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:15:14.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:15:44.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:15:44.238 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:16:14.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:16:14.344 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:16:44.496 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:16:44.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:17:14.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:17:14.639 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:17:44.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:17:44.810 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:18:14.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:18:14.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:18:45.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:18:45.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:19:15.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:19:15.258 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:19:45.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:19:45.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:20:15.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:20:15.563 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:20:45.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:20:45.636 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:21:15.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:21:15.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:21:45.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:21:45.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:22:15.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:22:15.853 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:22:46.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:22:46.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:23:16.185 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:23:16.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:23:46.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:23:46.335 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:24:16.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:24:16.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:24:46.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:24:46.632 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:25:16.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:25:16.785 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:25:46.933 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:25:46.936 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:26:17.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:26:17.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:26:47.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:26:47.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:27:17.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:27:17.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:27:47.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:27:47.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:28:17.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:28:17.493 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:28:47.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:28:47.624 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:29:17.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:29:17.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:29:47.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:29:47.859 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:30:17.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:30:17.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:30:48.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:30:48.147 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:31:18.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:31:18.268 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:31:48.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:31:48.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:32:18.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:32:18.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:32:48.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:32:48.686 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:33:18.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:33:18.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:33:48.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:33:48.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:34:19.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:34:19.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:34:49.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:34:49.223 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:35:19.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:35:19.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:35:49.501 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:35:49.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:36:19.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:36:19.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:36:49.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:36:49.808 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:37:19.977 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:37:19.980 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:37:50.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:37:50.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:38:20.293 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:38:20.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:38:50.428 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:38:50.431 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:39:20.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:39:20.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:39:50.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:39:50.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:40:20.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:40:20.855 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:40:51.034 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:40:51.037 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:41:21.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:41:21.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:41:51.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:41:51.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:42:21.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:42:21.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:42:51.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:42:51.676 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:43:21.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:43:21.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:43:51.933 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:43:51.936 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:44:22.069 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:44:22.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:44:52.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:44:52.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:45:22.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:45:22.359 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:45:52.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:45:52.512 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:46:22.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:46:22.656 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:46:52.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:46:52.796 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:47:22.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:47:22.976 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:47:53.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:47:53.140 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:48:23.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:48:23.287 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:48:53.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:48:53.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:49:23.545 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:49:23.548 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:49:53.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:49:53.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:50:23.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:50:23.844 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:50:53.997 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:50:54.000 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:51:24.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:51:24.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:51:54.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:51:54.292 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:52:24.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:52:24.432 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:52:54.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:52:54.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:53:24.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:53:24.685 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:53:54.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:53:54.813 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:54:24.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:54:24.958 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:54:55.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:54:55.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:55:25.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:55:25.219 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:55:55.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:55:55.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:56:25.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:56:25.543 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:56:55.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:56:55.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:57:25.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:57:25.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:57:55.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:57:55.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:58:26.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:58:26.099 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:58:56.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:58:56.243 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:59:26.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:59:26.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:59:56.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T05:59:56.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:00:26.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:00:26.690 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:00:56.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:00:56.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:01:26.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:01:26.962 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:01:57.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:01:57.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:02:27.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:02:27.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:02:57.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:02:57.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:03:27.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:03:27.547 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:03:57.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:03:57.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:04:27.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:04:27.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:04:58.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:04:58.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:05:28.150 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:05:28.153 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:05:58.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:05:58.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:06:28.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:06:28.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:06:58.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:06:58.584 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:07:28.717 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:07:28.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:07:58.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:07:58.880 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:08:29.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:08:29.007 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:08:59.129 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:08:59.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:09:29.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:09:29.262 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:09:59.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:09:59.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:10:29.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:10:29.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:10:59.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:10:59.693 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:11:29.842 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:11:29.845 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:12:00.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:12:00.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:12:30.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:12:30.282 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:13:00.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:13:00.409 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:13:30.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:13:30.527 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:14:00.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:14:00.688 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:14:30.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:14:30.807 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:15:00.947 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:15:00.950 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:15:31.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:15:31.077 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:16:01.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:16:01.228 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:16:31.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:16:31.395 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:17:01.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:17:01.550 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:17:31.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:17:31.695 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:18:01.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:18:01.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:18:32.012 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:18:32.015 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:19:02.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:19:02.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:19:32.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:19:32.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:20:02.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:20:02.414 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:20:32.550 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:20:32.553 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:21:02.756 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:21:02.759 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:21:32.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:21:32.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:22:02.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:22:02.995 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:22:33.064 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:22:33.067 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:23:03.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:23:03.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:23:33.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:23:33.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:24:03.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:24:03.271 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:24:33.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:24:33.407 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:25:03.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:25:03.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:25:33.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:25:33.611 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:26:03.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:26:03.732 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:26:33.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:26:33.879 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:27:04.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:27:04.018 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:27:34.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:27:34.165 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:28:04.293 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:28:04.296 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:28:34.415 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:28:34.418 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:29:04.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:29:04.565 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:29:34.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:29:34.762 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:30:04.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:30:04.893 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:30:34.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:30:34.999 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:31:05.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:31:05.130 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:31:35.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:31:35.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:32:05.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:32:05.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:32:35.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:32:35.530 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:33:05.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:33:05.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:33:35.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:33:35.825 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:34:05.944 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:34:05.947 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:34:36.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:34:36.090 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:35:06.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:35:06.226 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:35:36.367 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:35:36.370 DEBUG:teuthology.orchestra.run.smithi082:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-07-05T06:35:54.426 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-07-05T06:35:54.427 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2023-07-05T06:35:54.428 DEBUG:teuthology.task.console_log:Killing console logger for smithi082 2023-07-05T06:35:54.430 DEBUG:teuthology.exit:Finished running handlers